CN109962857B - Flow control method, flow control device and computer readable storage medium - Google Patents
Flow control method, flow control device and computer readable storage medium Download PDFInfo
- Publication number
- CN109962857B CN109962857B CN201711425947.7A CN201711425947A CN109962857B CN 109962857 B CN109962857 B CN 109962857B CN 201711425947 A CN201711425947 A CN 201711425947A CN 109962857 B CN109962857 B CN 109962857B
- Authority
- CN
- China
- Prior art keywords
- storage node
- flow
- time interval
- disk
- reconstructed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/20—Traffic policing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/29—Flow control; Congestion control using a combination of thresholds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/30—Flow control; Congestion control in combination with information about buffer occupancy at either end or at transit nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/70—Admission control; Resource allocation
- H04L47/80—Actions related to the user profile or the type of traffic
- H04L47/805—QOS or priority aware
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present disclosure provides a flow control method, apparatus and computer readable storage medium, which relate to the technical field of distributed storage, and the method includes: acquiring the service flow and the reconstruction flow of each storage node and the running state of a disk in each storage node in the current time interval; predicting the service flow of each storage node in the next time interval by using the service flow model of each storage node; predicting the reconstructed flow of each storage node in the next time interval by using the reconstructed flow model of each storage node; summing the predicted service flow and reconstruction flow of each storage node in the next time interval to obtain the total flow of all the storage nodes in the next time interval; and under the condition that the total flow is greater than the preset total flow upper limit, determining a bottleneck storage node according to the predicted service flow and the predicted reconstruction flow of each storage node in the next time interval, and performing current limiting control on a magnetic disk with the reconstruction flow in the bottleneck storage node.
Description
Technical Field
The present disclosure relates to the field of distributed storage technologies, and in particular, to a flow control method, apparatus, and computer-readable storage medium.
Background
The distributed block storage mainstream adopts an X86 server as a data storage node, and data is stored on a disk in the storage node, such as a SATA disk.
Typically, after a period of use, e.g., 2-3 years, a large number of disk failures occur, resulting in a larger scale reconstruction of the data, i.e., copying the data on the failed disk of one node to the robust disk of another node. The data reconstruction can generate more read-write operations and more reconstruction flow, and the reconstruction flow can occupy the normal service flow of the storage node, so that the service operation of the storage node is influenced.
Disclosure of Invention
One object of the present disclosure is: and the adverse effect of the reconstructed flow on the service operation of the bottleneck storage node is reduced.
According to an aspect of the embodiments of the present disclosure, there is provided a flow control method, including: acquiring the service flow and the reconstruction flow of each storage node and the running state of a disk in each storage node in the current time interval; predicting the service flow of each storage node in the next time interval by using the service flow model of each storage node according to the service flow of each storage node in the current time interval and the running state of a magnetic disk in each storage node; predicting the reconstruction flow of each storage node in the next time interval by using the reconstruction flow model of each storage node according to the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node; summing the predicted service flow and reconstruction flow of each storage node in the next time interval to obtain the total flow of all the storage nodes in the next time interval; and under the condition that the total flow is greater than the preset total flow upper limit, determining a bottleneck storage node according to the predicted service flow and reconstructed flow of each storage node in the next time interval, and performing flow limiting control on a magnetic disc with the reconstructed flow in the bottleneck storage node.
In some embodiments, the method further comprises: acquiring service flow and reconstructed flow of each storage node in different time intervals and the running state of a disk in each storage node; taking the service flow of each storage node in the previous time interval of two adjacent time intervals, the running state of a disk in each storage node and the service flow in the later time interval as training samples, and training the service flow model of each storage node, so that the service flow model of each storage node can predict the service flow of each storage node in the next time interval according to the service flow of each storage node in the current time interval and the running state of a disk in each storage node; and taking the reconstructed flow of each storage node in the previous time interval of the two adjacent time intervals, the running state of the disk in each storage node and the reconstructed flow in the later time interval as training samples, and training the reconstructed flow model of each storage node, so that the reconstructed flow model of each storage node can predict the reconstructed flow of each storage node in the next time interval according to the reconstructed flow of each storage node in the current time interval and the running state of the disk in each storage node.
In some embodiments, before summing the traffic flow and the reconstructed flow of each storage node in the predicted next time interval, the method further includes: adjusting the read-write rate of a disk with reconstructed flow in the storage node and the reconstructed flow of the storage node in the current time interval according to the sum of the predicted service flow and the reconstructed flow of the storage node in the next time interval; and according to the adjusted reconstructed flow of the storage node in the current time interval and the running state of the disk in the storage node, predicting the reconstructed flow of the storage node in the next time interval again by using the reconstructed flow model of the storage node.
In some embodiments, the throttling the disk with the reconstructed flow in the bottleneck storage node comprises: redirecting a second disk in the bottleneck storage node, which is writing data to the first disks of the other storage nodes, to write data to a third disk in the bottleneck storage node, and forbidding the disks of the other storage nodes from writing data to the disk of the bottleneck storage node.
In some embodiments, the method further comprises: and writing the flow written from the second disk to the third disk into the first disk under the condition that the total flow is smaller than a preset total flow lower limit, and recovering the disks of other storage nodes to write data to the disk of the bottleneck storage node.
According to another aspect of the embodiments of the present disclosure, there is provided a flow control device including: the first acquisition module is used for acquiring the service flow and the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node; the first prediction module is used for predicting the service flow of each storage node in the next time interval by using the service flow model of each storage node according to the service flow of each storage node in the current time interval and the running state of a disk in each storage node; the second prediction module is used for predicting the reconstruction flow of each storage node in the next time interval by using the reconstruction flow model of each storage node according to the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node; the summation module is used for summing the predicted service flow and the reconstructed flow of each storage node in the next time interval so as to obtain the total flow of all the storage nodes in the next time interval; and the control module is used for determining a bottleneck storage node according to the predicted service flow and the reconstructed flow of each storage node in the next time interval under the condition that the total flow is greater than the preset total flow upper limit, and performing flow limiting control on a magnetic disk with the reconstructed flow in the bottleneck storage node.
In some embodiments, the apparatus further comprises: the second acquisition module is used for acquiring the service flow and the reconstruction flow of each storage node in different time intervals and the running state of a magnetic disk in each storage node; the first training module is used for training a service flow model of each storage node by taking the service flow of each storage node in the previous time interval of two adjacent time intervals, the running state of a disk in each storage node and the service flow in the later time interval as training samples, so that the service flow model of each storage node can predict the service flow of each storage node in the next time interval according to the service flow of each storage node in the current time interval and the running state of the disk in each storage node; the second training module is used for training the reconstructed flow model of each storage node by taking the reconstructed flow of each storage node in the previous time interval of the two adjacent time intervals, the running state of the disk in each storage node and the reconstructed flow in the later time interval as training samples, so that the reconstructed flow model of each storage node can predict the reconstructed flow of each storage node in the next time interval according to the reconstructed flow of each storage node in the current time interval and the running state of the disk in each storage node.
In some embodiments, the apparatus further comprises: the adjusting module is used for adjusting the read-write speed of a disk with reconstructed flow in the storage node and the reconstructed flow of the storage node in the current time interval according to the sum of the predicted service flow and the reconstructed flow of the storage node in the next time interval; and the second prediction module is used for predicting the reconstruction flow of the storage node in the next time interval again by using the reconstruction flow model of the storage node according to the adjusted reconstruction flow of the storage node in the current time interval and the running state of a disk in the storage node.
In some embodiments, the control module is configured to redirect a second disk in the bottleneck storage node that is writing data to the first disk of the other storage node to write data to a third disk in the bottleneck storage node, and prohibit the disks of the other storage nodes from writing data to the disk of the bottleneck storage node.
In some embodiments, the control module is further configured to, when the total traffic is smaller than a preset total lower traffic limit, write the traffic written from the second disk to the third disk to the first disk, and recover the disks of the other storage nodes to write data to the disk of the bottleneck storage node.
According to still another aspect of the embodiments of the present disclosure, there is provided a flow control device including: a memory; and a processor coupled to the memory, the processor configured to perform the method of any of the above embodiments based on instructions stored in the memory.
According to yet another aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method according to any one of the embodiments described above.
In the embodiment of the disclosure, according to the service flow and the reconstruction flow of each storage node in the current time interval and the operating state of the disk in each storage node, the service flow and the reconstruction flow of each storage node in the next time interval can be predicted by using the service flow model and the reconstruction flow model. And then, carrying out current limiting control on a disk with reconstructed flow in the bottleneck storage node which is about to generate the conflict between the reconstructed flow and the service flow so as to reduce the adverse effect of the reconstructed flow on the service operation of the bottleneck storage node.
The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.
Drawings
In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a schematic flow diagram of a flow control method according to one embodiment of the present disclosure;
FIG. 2 is a flow diagram of a flow control method according to another embodiment of the present disclosure;
FIG. 3 is a flow chart diagram of a flow control method according to yet another embodiment of the present disclosure;
FIG. 4 is a schematic illustration of a flow control device according to one embodiment of the present disclosure;
FIG. 5 is a schematic illustration of a flow control device according to another embodiment of the present disclosure;
FIG. 6 is a schematic structural view of a flow control device according to yet another embodiment of the present disclosure;
FIG. 7 is a schematic structural view of a flow control device according to yet another embodiment of the present disclosure;
FIG. 8 is a schematic view of a flow control device according to yet another embodiment of the present disclosure.
Detailed Description
The technical solutions in the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the embodiments described are only some embodiments of the present disclosure, rather than all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.
The relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.
Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.
In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.
Fig. 1 is a flow chart diagram of a flow control method according to one embodiment of the present disclosure.
In step 102, the service traffic of each storage node in the current time interval, the reconstruction traffic and the running state of the disk in each storage node are collected.
For example, a day may be divided into different time intervals according to a preset time interval, and the adjacent time intervals include a previous time interval and a next time interval. For the current time interval, the latter time interval is the next time interval. The predetermined time interval may be 2-4 hours, for example, 0-2 points may be used as one time interval, and 2-4 points may be used as the next time interval.
Multiple disks may be included within each storage node. The operating state of each disk may reflect the health of the disk, such as whether the disk is damaged, whether the disk writes data to other disks, whether the disk is written by other disks, and the like.
In step 104, according to the service traffic of each storage node in the current time interval and the running state of the disk in each storage node, the service traffic of each storage node in the next time interval is predicted by using the service traffic model of each storage node. Here, the traffic flow model may be a linear regression model, for example.
The traffic flow model of each storage node can be obtained as follows: the method comprises the steps that the service flow of each storage node in different time intervals and the running state of a magnetic disk in each storage node are collected in advance; then, the service flow of each storage node in the previous time interval, the running state of the disk in each storage node and the service flow in the later time interval of two adjacent time intervals are taken as training samples, and the service flow model of each storage node is trained, so that the service flow model of each storage node can predict the service flow of each storage node in the next time interval according to the service flow of each storage node in the current time interval and the running state of the disk in each storage node.
In step 106, according to the reconstructed flow of each storage node in the current time interval and the running state of the disk in each storage node, the reconstructed flow model of each storage node is used for predicting the reconstructed flow of each storage node in the next time interval. Here, the reconstructed flow rate model may be a linear regression model, for example.
The traffic flow model of each storage node can be obtained as follows: acquiring the reconstructed flow of each storage node in different time intervals and the running state of a magnetic disk in each storage node in advance; then, the reconstructed flow of each storage node in the previous time interval, the running state of the disk in each storage node and the reconstructed flow in the later time interval of two adjacent time intervals are taken as training samples, and the reconstructed flow model of each storage node is trained, so that the reconstructed flow model of each storage node can predict the reconstructed flow of each storage node in the next time interval according to the reconstructed flow of each storage node in the current time interval and the running state of the disk in each storage node.
In step 108, the predicted traffic flow and reconstructed flow of each storage node in the next time interval are summed to obtain a total flow of all storage nodes in the next time interval.
In step 110, when the total flow is greater than the preset total flow upper limit, determining a bottleneck storage node according to the predicted traffic flow and reconstructed flow of each storage node in the next time interval, and performing flow limiting control on a disk with reconstructed flow in the bottleneck storage node.
Here, the preset total flow upper limit may be set according to the total flow allowed by the network. Under the condition that the sum of the service flow and the reconstruction flow of the storage node in the next time interval is larger than a preset threshold value, the storage node can be determined to be a bottleneck storage node, namely, in the next time interval, the service flow and the reconstruction flow of the storage node are in a contention phenomenon.
The following describes a specific implementation of flow restriction control for a disk with reconstructed flow in a bottleneck storage node.
As a specific implementation manner, a second disk in the bottleneck storage node, which is writing data to the first disk of the other storage node, may be redirected to write data to a third disk in the bottleneck storage node, that is, reconstructed traffic flowing from the bottleneck node is limited in the bottleneck node. Accordingly, in the case that the total flow is smaller than the preset total flow lower limit, the data written from the second disk to the third disk may be written to the first disk, that is, the normal reconfiguration operation is resumed.
As another specific implementation, the disks of the other storage nodes may be prohibited from writing data to the disk of the bottleneck storage node, that is, the reconstructed traffic flowing from the other storage nodes to the bottleneck node is prohibited. Correspondingly, under the condition that the total flow is smaller than the preset total flow lower limit, the disks of the other storage nodes are recovered to write data into the disk of the bottleneck storage node, that is, the normal reconstruction operation is recovered.
As another specific implementation manner, a second disk in the bottleneck storage node, which is writing data to the first disk of the other storage node, may be redirected to write data to a third disk in the bottleneck storage node, and the disks of the other storage nodes are prohibited from writing data to the disk of the bottleneck storage node.
In the above embodiment, according to the service traffic, the reconstructed traffic, and the operating state of the disk in each storage node in the current time interval, the service traffic and the reconstructed traffic of each storage node in the next time interval may be predicted by using the service traffic model and the reconstructed traffic model. And then, carrying out current limiting control on a disk with reconstructed flow in the bottleneck storage node which is about to generate the conflict between the reconstructed flow and the service flow so as to reduce the adverse effect of the reconstructed flow on the service operation of the bottleneck storage node.
Fig. 2 is a flow chart diagram of a flow control method according to another embodiment of the present disclosure.
In step 202, the service traffic, the reconstruction traffic and the running state of the disk in each storage node in different time intervals are collected.
In step 204, the service traffic of each storage node in the previous time interval and the operating state of the disk in each storage node in the two adjacent time intervals, and the service traffic in the subsequent time interval are taken as training samples, and the service traffic model of each storage node is trained, so that the service traffic model of each storage node can predict the service traffic of each storage node in the next time interval according to the service traffic of each storage node in the current time interval and the operating state of the disk in each storage node.
In step 206, the reconstructed traffic of each storage node in the previous time interval of the two adjacent time intervals, the operating state of the disk in each storage node, and the reconstructed traffic in the next time interval are taken as training samples, and the reconstructed traffic model of each storage node is trained, so that the reconstructed traffic model of each storage node can predict the reconstructed traffic of each storage node in the next time interval according to the reconstructed traffic of each storage node in the current time interval and the operating state of the disk in each storage node.
In step 208, the traffic flow, the reconstruction flow and the running state of the disk in each storage node in the current time interval are collected.
In step 210, according to the service traffic of each storage node in the current time interval and the operating state of the disk in each storage node, the service traffic of each storage node in the next time interval is predicted by using the service traffic model of each storage node.
In step 212, according to the reconstructed traffic of each storage node in the current time interval and the running state of the disk in each storage node, the reconstructed traffic model of each storage node is used to predict the reconstructed traffic of each storage node in the next time interval.
In step 214, the predicted traffic flow and reconstructed flow of each storage node in the next time interval are summed to obtain the total flow of all storage nodes in the next time interval.
In step 216, in a case that the total flow is greater than the preset total flow upper limit, determining a bottleneck storage node according to the predicted service flow and reconstructed flow of each storage node in the next time interval, and performing flow limiting control on a disk with reconstructed flow in the bottleneck storage node.
The specific implementation of steps 208-216 can be found in reference to the above description.
In the above embodiment, the service traffic model and the reconstructed traffic model are obtained first by training. And predicting the service flow and the reconstruction flow of each storage node in the next time interval by using the service flow model and the reconstruction flow model according to the service flow and the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node. And then, the current limiting control is carried out on the disk with the reconstructed flow in the bottleneck storage node which is about to generate the conflict between the reconstructed flow and the service flow, so that the adverse effect of the reconstructed flow on the service operation of the bottleneck storage node is reduced.
Fig. 3 is a flow chart illustrating a flow control method according to another embodiment of the present disclosure.
In step 302, the service traffic, the reconstructed traffic and the running state of the disk in each storage node in the current time interval are collected.
In step 304, the service traffic of each storage node in the next time interval is predicted by using the service traffic model of each storage node according to the service traffic of each storage node in the current time interval and the operating state of the disk in each storage node.
In step 306, according to the reconstructed traffic of each storage node in the current time interval and the running state of the disk in each storage node, the reconstructed traffic model of each storage node is used to predict the reconstructed traffic of each storage node in the next time interval.
In step 308, the read-write rate of the disk with the reconstructed traffic in the storage node and the reconstructed traffic in the current time interval are adjusted according to the sum of the predicted service traffic and reconstructed traffic of the storage node in the next time interval.
For example, if the sum of the service traffic and the reconstructed traffic of the storage node in the next time interval is greater than the preset threshold, the read-write rate of the disk with the reconstructed traffic in the storage node may be increased, and the reconstructed traffic of the storage node in the current time interval may be decreased.
For another example, if the sum of the service traffic and the reconstructed traffic of the storage node in the next time interval is smaller than the preset threshold, the read-write rate of the disk with the reconstructed traffic in the storage node may be decreased, and the reconstructed traffic of the storage node in the current time interval may be increased.
In step 310, according to the adjusted reconstructed flow of the storage node in the current time interval and the running state of the disk in the storage node, the reconstructed flow of the storage node in the next time interval is predicted again by using the reconstructed flow model of the storage node.
Therefore, the service flow and the reconstruction flow of each storage node in the next time interval can be obtained.
In step 312, the predicted traffic flow and reconstructed flow of each storage node in the next time interval are summed to obtain a total flow of all storage nodes in the next time interval.
It should be understood that, for a storage node that predicts the reconstructed traffic again, the reconstructed traffic of the storage node in the next time interval is the reconstructed traffic after the prediction is performed again.
In step 314, when the total flow is greater than the preset total flow upper limit, determining a bottleneck storage node according to the predicted traffic flow and reconstructed flow of each storage node in the next time interval, and performing flow limiting control on a disk with reconstructed flow in the bottleneck storage node.
In the above embodiment, after the service traffic and the reconstructed traffic of the storage node in the next time interval are predicted, the read-write rate of the disk with the reconstructed traffic in the storage node in the current time interval and the reconstructed traffic of the storage node in the current time interval may be adjusted according to the sum of the predicted service traffic and the reconstructed traffic, so that on one hand, the bandwidth may be fully utilized, and on the other hand, the reconstruction may be completed as soon as possible on the premise of ensuring the service traffic, so as to further reduce adverse effects on the service operation of the subsequently determined bottleneck storage node.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the device embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
FIG. 4 is a schematic diagram of a flow control device according to one embodiment of the present disclosure. As shown in fig. 4, the apparatus of this embodiment includes:
a first acquisition module 401, configured to acquire a service traffic, a reconstruction traffic, and an operation state of a disk in each storage node in a current time interval;
a first prediction module 402, configured to predict, according to the service traffic of each storage node in the current time interval and the operating state of the disk in each storage node, the service traffic of each storage node in the next time interval by using a service traffic model of each storage node;
a second prediction module 403, configured to predict a reconstructed flow of each storage node in a next time interval by using a reconstructed flow model of each storage node according to the reconstructed flow of each storage node in the current time interval and an operating state of a disk in each storage node;
a summing module 404, configured to sum the predicted service traffic and reconstructed traffic of each storage node in the next time interval to obtain a total traffic of all storage nodes in the next time interval;
and the control module 405 is configured to, when the total flow is greater than the preset total flow upper limit, determine a bottleneck storage node according to the predicted service flow and reconstructed flow of each storage node in the next time interval, and perform flow limiting control on a disk with the reconstructed flow in the bottleneck storage node.
As a specific implementation manner, the control module 405 may be configured to redirect a second disk in the bottleneck storage node, which is writing data to the first disk of the other storage node, to write data to a third disk in the bottleneck storage node, and prohibit the disks of the other storage nodes from writing data to the disk of the bottleneck storage node. Further, the control module 405 may be further configured to, when the total traffic is smaller than the preset total lower traffic limit, write the traffic written from the second disk to the third disk to the first disk, and restore the disks of the other storage nodes to write data to the disk of the bottleneck storage node.
FIG. 5 is a schematic diagram of a flow control device according to another embodiment of the present disclosure. As shown in fig. 5, the apparatus of this embodiment further includes, compared to fig. 4:
a second collecting module 501, configured to collect service traffic and reconstructed traffic of each storage node in different time intervals, and an operating state of a disk in each storage node;
a first training module 502, configured to train a service traffic model of each storage node by using a service traffic of each storage node in a previous time interval and an operating state of a disk in each storage node in two adjacent time intervals, and a service traffic in a next time interval as training samples, so that the service traffic model of each storage node can predict a service traffic of each storage node in a next time interval according to the service traffic of each storage node in a current time interval and the operating state of a disk in each storage node;
the second training module 503 is configured to train the reconstructed traffic model of each storage node with the reconstructed traffic of each storage node in the previous time interval and the operating state of the disk in each storage node in the two adjacent time intervals and the reconstructed traffic of each storage node in the subsequent time interval as training samples, so that the reconstructed traffic model of each storage node can predict the reconstructed traffic of each storage node in the next time interval according to the reconstructed traffic of each storage node in the current time interval and the operating state of the disk in each storage node.
FIG. 6 is a schematic structural view of a flow control device according to yet another embodiment of the present disclosure. As shown in fig. 6, compared with fig. 4, the apparatus in this embodiment further includes an adjusting module 601, configured to adjust a reconstructed traffic of the storage node in a current time interval according to a sum of the predicted traffic and the reconstructed traffic of the storage node in a next time interval. Correspondingly, the second prediction module 403 in this embodiment is configured to predict the reconstructed traffic of the storage node in the next time interval again by using the reconstructed traffic model of the storage node according to the adjusted reconstructed traffic of the storage node in the current time interval and the operating state of the disk in the storage node.
FIG. 7 is a schematic diagram of a flow control device according to yet another embodiment of the present disclosure. As shown in fig. 7, the apparatus of this embodiment includes a memory 701 and a processor 702. The memory 701 may be a magnetic disk, flash memory, or any other non-volatile storage medium. The memory 701 is used for storing instructions corresponding to the method of any one of the foregoing embodiments. Coupled to memory 701 is a processor 702, which may be implemented as one or more integrated circuits, such as a microprocessor or microcontroller. The processor 702 is configured to execute instructions stored in the memory 701.
FIG. 8 is a schematic view of a flow control device according to yet another embodiment of the present disclosure. As shown in fig. 8, the apparatus 800 of this embodiment includes a memory 801 and a processor 802. The processor 802 is coupled to the memory 801 by a BUS (BUS) 803. The device 800 may also be connected to an external storage device 805 through a storage interface 804 in order to call up external data, and may also be connected to a network or an external computer device (not shown) through a network interface 806.
The present disclosure also provides a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the method of any of the preceding embodiments.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, apparatus, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Thus far, the present disclosure has been described in detail. Some details that are well known in the art have not been described in order to avoid obscuring the concepts of the present disclosure. It will be fully apparent to those skilled in the art from the foregoing description how to practice the presently disclosed embodiments.
Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the foregoing examples are for purposes of illustration only and are not intended to limit the scope of the present disclosure. It will be appreciated by those skilled in the art that modifications may be made to the above embodiments without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.
Claims (12)
1. A method of flow control, comprising:
acquiring the service flow and the reconstruction flow of each storage node and the running state of a disk in each storage node in the current time interval;
predicting the service flow of each storage node in the next time interval by using the service flow model of each storage node according to the service flow of each storage node in the current time interval and the running state of a magnetic disk in each storage node;
predicting the reconstruction flow of each storage node in the next time interval by using the reconstruction flow model of each storage node according to the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node;
summing the predicted service flow and reconstruction flow of each storage node in the next time interval to obtain the total flow of all the storage nodes in the next time interval;
when the total flow is greater than a preset total flow upper limit, determining a bottleneck storage node according to the predicted service flow and reconstructed flow of each storage node in the next time interval, and performing flow limiting control on a disk with the reconstructed flow in the bottleneck storage node, wherein the flow limiting control method comprises the following steps: redirecting a second disk, which is writing data to the first disks of other storage nodes, in the bottleneck storage node to write data to a third disk in the bottleneck storage node;
and writing the flow written from the second disk to the third disk into the first disk under the condition that the total flow is smaller than a preset total flow lower limit.
2. The method of claim 1, further comprising:
acquiring service flow and reconstructed flow of each storage node in different time intervals and the running state of a disk in each storage node;
taking the service flow of each storage node in the previous time interval of two adjacent time intervals, the running state of a disk in each storage node and the service flow in the later time interval as training samples, and training the service flow model of each storage node, so that the service flow model of each storage node can predict the service flow of each storage node in the next time interval according to the service flow of each storage node in the current time interval and the running state of a disk in each storage node;
and taking the reconstructed flow of each storage node in the previous time interval of the two adjacent time intervals, the running state of the disk in each storage node and the reconstructed flow in the later time interval as training samples, and training the reconstructed flow model of each storage node, so that the reconstructed flow model of each storage node can predict the reconstructed flow of each storage node in the next time interval according to the reconstructed flow of each storage node in the current time interval and the running state of the disk in each storage node.
3. The method of claim 1, wherein prior to summing the traffic flow and the reconstructed flow for each storage node for the predicted next time interval, further comprising:
adjusting the read-write rate of a disk with the reconstructed flow in the storage node and the reconstructed flow of the storage node in the current time interval according to the sum of the predicted service flow and the reconstructed flow of the storage node in the next time interval;
and according to the adjusted reconstructed flow of the storage node in the current time interval and the running state of the disk in the storage node, predicting the reconstructed flow of the storage node in the next time interval again by using the reconstructed flow model of the storage node.
4. The method of any of claims 1-3, wherein the throttling control of the disk with reconstructed traffic within the bottleneck storage node further comprises:
forbidding the disks of other storage nodes to write data into the disk of the bottleneck storage node;
the method further comprises the following steps:
and under the condition that the total flow is smaller than the preset total flow lower limit, restoring the disks of other storage nodes to write data into the disk of the bottleneck storage node.
5. The method of claim 3, wherein adjusting the read-write rate of the disk with the reconstructed traffic in the storage node and the reconstructed traffic in the current time interval according to the predicted sum of the service traffic and the reconstructed traffic of the storage node in the next time interval comprises:
if the sum of the service flow and the reconstructed flow of the storage node in the next time interval is larger than a preset threshold, increasing the read-write speed of a disk with the reconstructed flow in the storage node, and decreasing the reconstructed flow of the storage node in the current time interval;
and if the sum of the service flow and the reconstructed flow of the storage node in the next time interval is smaller than a preset threshold, reducing the read-write speed of a disk with the reconstructed flow in the storage node, and increasing the reconstructed flow of the storage node in the current time interval.
6. A flow control device comprising:
the first acquisition module is used for acquiring the service flow and the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node;
the first prediction module is used for predicting the service flow of each storage node in the next time interval by using the service flow model of each storage node according to the service flow of each storage node in the current time interval and the running state of a disk in each storage node;
the second prediction module is used for predicting the reconstruction flow of each storage node in the next time interval by using the reconstruction flow model of each storage node according to the reconstruction flow of each storage node in the current time interval and the running state of a disk in each storage node;
the summation module is used for summing the predicted service flow and the reconstructed flow of each storage node in the next time interval so as to obtain the total flow of all the storage nodes in the next time interval;
the control module is used for determining bottleneck storage nodes according to the predicted service flow and the reconstructed flow of each storage node in the next time interval under the condition that the total flow is greater than the preset total flow upper limit, and performing flow limiting control on a magnetic disk with the reconstructed flow in the bottleneck storage nodes;
the control module is used for redirecting a second disk, which is writing data to the first disks of other storage nodes in the bottleneck storage node, to a third disk in the bottleneck storage node; and writing the flow written from the second disk to the third disk to the first disk under the condition that the total flow is less than a preset total flow lower limit.
7. The apparatus of claim 6, further comprising:
the second acquisition module is used for acquiring the service flow and the reconstruction flow of each storage node in different time intervals and the running state of a disk in each storage node;
the first training module is used for training a service flow model of each storage node by taking the service flow of each storage node in the previous time interval and the running state of a disk in each storage node in two adjacent time intervals and the service flow of each storage node in the later time interval as training samples, so that the service flow model of each storage node can predict the service flow of each storage node in the next time interval according to the service flow of each storage node in the current time interval and the running state of the disk in each storage node;
the second training module is used for training the reconstructed flow model of each storage node by taking the reconstructed flow of each storage node in the previous time interval of the two adjacent time intervals, the running state of the disk in each storage node and the reconstructed flow in the later time interval as training samples, so that the reconstructed flow model of each storage node can predict the reconstructed flow of each storage node in the next time interval according to the reconstructed flow of each storage node in the current time interval and the running state of the disk in each storage node.
8. The apparatus of claim 6, further comprising:
the adjusting module is used for adjusting the read-write speed of a disk with reconstructed flow in the storage node and the reconstructed flow of the storage node in the current time interval according to the sum of the predicted service flow and the reconstructed flow of the storage node in the next time interval;
and the second prediction module is used for predicting the reconstruction flow of the storage node in the next time interval again by using the reconstruction flow model of the storage node according to the adjusted reconstruction flow of the storage node in the current time interval and the running state of a disk in the storage node.
9. The apparatus of any one of claims 6-8,
the control module is also used for forbidding the disks of other storage nodes to write data into the disk of the bottleneck storage node; and under the condition that the total flow is smaller than the preset total flow lower limit, restoring the disks of other storage nodes to write data into the disk of the bottleneck storage node.
10. The apparatus of claim 8, wherein,
the adjusting module is used for increasing the read-write speed of a disk with reconstructed flow in the storage node and decreasing the reconstructed flow of the storage node in the current time interval if the sum of the service flow and the reconstructed flow of the storage node in the next time interval is greater than a preset threshold; and if the sum of the service flow and the reconstructed flow of the storage node in the next time interval is less than a preset threshold, reducing the read-write speed of a disk with the reconstructed flow in the storage node, and increasing the reconstructed flow of the storage node in the current time interval.
11. A flow control device comprising:
a memory; and
a processor coupled to the memory, the processor configured to perform the method of any of claims 1-5 based on instructions stored in the memory.
12. A computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711425947.7A CN109962857B (en) | 2017-12-26 | 2017-12-26 | Flow control method, flow control device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711425947.7A CN109962857B (en) | 2017-12-26 | 2017-12-26 | Flow control method, flow control device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109962857A CN109962857A (en) | 2019-07-02 |
| CN109962857B true CN109962857B (en) | 2022-08-30 |
Family
ID=67021452
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711425947.7A Active CN109962857B (en) | 2017-12-26 | 2017-12-26 | Flow control method, flow control device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109962857B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113381941B (en) * | 2020-02-25 | 2022-09-30 | 深信服科技股份有限公司 | Task scheduling method and device, electronic equipment and computer storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102981778A (en) * | 2012-11-15 | 2013-03-20 | 浙江宇视科技有限公司 | Redundant array of independent disks (RAID) array reconstruction method and device thereof |
| CN103488434A (en) * | 2013-09-23 | 2014-01-01 | 浪潮电子信息产业股份有限公司 | Method for improving disk array reliability |
| US9087012B1 (en) * | 2014-06-04 | 2015-07-21 | Pure Storage, Inc. | Disaster recovery at high reliability in a storage cluster |
| CN105589795A (en) * | 2014-12-31 | 2016-05-18 | 中国银联股份有限公司 | Disk failure prediction method and device based on prediction model |
| CN106843765A (en) * | 2017-01-22 | 2017-06-13 | 郑州云海信息技术有限公司 | A kind of disk management method and device |
| CN107479836A (en) * | 2017-08-29 | 2017-12-15 | 郑州云海信息技术有限公司 | Disk failure monitoring method, device and storage system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8756310B2 (en) * | 2011-03-09 | 2014-06-17 | International Business Machines Corporation | Comprehensive bottleneck detection in a multi-tier enterprise storage system |
-
2017
- 2017-12-26 CN CN201711425947.7A patent/CN109962857B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102981778A (en) * | 2012-11-15 | 2013-03-20 | 浙江宇视科技有限公司 | Redundant array of independent disks (RAID) array reconstruction method and device thereof |
| CN103488434A (en) * | 2013-09-23 | 2014-01-01 | 浪潮电子信息产业股份有限公司 | Method for improving disk array reliability |
| US9087012B1 (en) * | 2014-06-04 | 2015-07-21 | Pure Storage, Inc. | Disaster recovery at high reliability in a storage cluster |
| CN105589795A (en) * | 2014-12-31 | 2016-05-18 | 中国银联股份有限公司 | Disk failure prediction method and device based on prediction model |
| CN106843765A (en) * | 2017-01-22 | 2017-06-13 | 郑州云海信息技术有限公司 | A kind of disk management method and device |
| CN107479836A (en) * | 2017-08-29 | 2017-12-15 | 郑州云海信息技术有限公司 | Disk failure monitoring method, device and storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109962857A (en) | 2019-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9367598B2 (en) | Merging an out of synchronization indicator and a change recording indicator in response to a failure in consistency group formation | |
| CN107735767B (en) | Apparatus and method for virtual machine migration | |
| US10210053B2 (en) | Snapshots at real time intervals on asynchronous data replication system | |
| CN104714878B (en) | A kind of method and device of collector journal data | |
| CN107145403A (en) | The relevant database data retrogressive method of web oriented development environment | |
| US20130111172A1 (en) | Data Migration Between Storage Devices | |
| CN103810062A (en) | Slow disk detection method and device | |
| CN103389926B (en) | A kind of method and apparatus for backing up virtual disk | |
| CN105359108A (en) | Storage systems with adaptive erasure code generation | |
| CN110597655B (en) | Migration and erasure code-based reconstruction coupling rapid prediction repair method and device | |
| CN106100901B (en) | Flow velocity control method and device | |
| WO2013181185A1 (en) | Systems and methods for disaster recovery of multi-tier applications | |
| WO2016183918A1 (en) | Method and device for processing storage space | |
| CN112860968A (en) | Abnormity detection method and device | |
| CN111858108A (en) | Hard disk fault prediction method and device, electronic equipment and storage medium | |
| US11196624B2 (en) | Method and system for managing virtual datacenters | |
| CN106899654A (en) | A kind of sequence value generation method, apparatus and system | |
| CN106547607B (en) | Dynamic migration method and device for virtual machine | |
| CN109962857B (en) | Flow control method, flow control device and computer readable storage medium | |
| CN110532755B (en) | Computer-implemented risk identification method and device | |
| US20150112946A1 (en) | Computing device and data recovery method for distributed file system | |
| US9922149B2 (en) | Integration of functional analysis and common path pessimism removal in static timing analysis | |
| CN108536542A (en) | Position monitoring method and device | |
| CN112486731B (en) | Method, device, equipment and product for restoring distribution of placement groups | |
| CN109039695B (en) | Service fault processing method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |