CN111585904B - Bidirectional coflow scheduling system based on prediction of concurrent connection number - Google Patents

Bidirectional coflow scheduling system based on prediction of concurrent connection number Download PDF

Info

Publication number
CN111585904B
CN111585904B CN202010360227.2A CN202010360227A CN111585904B CN 111585904 B CN111585904 B CN 111585904B CN 202010360227 A CN202010360227 A CN 202010360227A CN 111585904 B CN111585904 B CN 111585904B
Authority
CN
China
Prior art keywords
host
connections
reducer
concurrent
connection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010360227.2A
Other languages
Chinese (zh)
Other versions
CN111585904A (en
Inventor
田臣
秦柳兰
傅晓明
江伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Hande Digital Twin Technology Research Institute Co ltd
Original Assignee
Nanjing Hande Digital Twin Technology Research Institute Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Hande Digital Twin Technology Research Institute Co ltd filed Critical Nanjing Hande Digital Twin Technology Research Institute Co ltd
Priority to CN202010360227.2A priority Critical patent/CN111585904B/en
Publication of CN111585904A publication Critical patent/CN111585904A/en
Application granted granted Critical
Publication of CN111585904B publication Critical patent/CN111585904B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/12Avoiding congestion; Recovering from congestion
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/12Avoiding congestion; Recovering from congestion
    • H04L47/127Avoiding congestion; Recovering from congestion by using congestion prediction
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/50Queue scheduling
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00Packet switching elements
    • H04L49/90Buffering arrangements
    • H04L49/9005Buffering arrangements using dynamic buffer space allocation
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D30/00Reducing energy consumption in communication networks
    • Y02D30/50Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The invention discloses a bidirectional coflow scheduling method based on the prediction of the number of concurrent connections, which comprises the following steps: s1, predicting the optimal number of concurrent connections on each host; s2, collecting information of the mapper and the retarder by adopting a global scheduling program, and generating a global candidate list on the tracker; s3, determining the number of concurrent connections of each reducer on the local scheduling program according to the dynamic priority list; s4, readjusting the connection status of each retarder on the micro-scheduler and reporting the current connection status to the global scheduler. The invention can predict the optimal number of concurrent connections, and avoids congestion caused by too many connections and bandwidth waste caused by too few connections. Meanwhile, the invention can combine the advantages of the two scheduling methods of the sender driver and the receiver driver to achieve the near-optimal scheduling effect.

Description

Bidirectional coflow scheduling system based on prediction of concurrent connection number
Technical Field
The invention relates to the technical field of data center network communication, in particular to a bidirectional coflow scheduling method based on the prediction of the number of concurrent connections.
Background
In data center networks, mainstream data parallel frameworks such as Hadoop and Spark are highly structured in network communication. Therefore, network level optimizations such as minimizing individual flow completion time or improving fairness between flows cannot improve performance. The flow is a network abstraction proposed for application level semantics, representing a set of related flows of the communication phase. Its completion requires that the last of all these streams has been completed. This abstraction shortens the gap between application-level semantics and network-level optimization. Minimizing the flow Completion Time (CCT) reduces the Completion Time of the corresponding job.
However, the latest coflow scheduling method has several disadvantages. On the one hand, they do not achieve optimal scheduling. Sender-driven coflow scheduling may be suboptimal when the bandwidth of the host ingress link becomes a bottleneck. Accordingly, receiver-driven scheduling may also be suboptimal when the bandwidth of the host egress link becomes a bottleneck. In a real network, these two situations often coexist due to the imbalance of task allocation. On the other hand, existing work cannot optimize the number of concurrent connections on each host. Good throughput has proven to be strongly correlated with the number of concurrent connections. For each host access link corresponding switch egress port, the queue length increases as the number of concurrent connections increases. When the limit of the queue buffer is exceeded, the switch begins to drop packets, resulting in retransmissions or even reconnection. The CCT may be extended due to too many or too few concurrent connections.
Disclosure of Invention
The invention aims to provide a bidirectional coflow scheduling method based on the prediction of the number of concurrent connections, which can predict the optimal number of the concurrent connections and avoid congestion caused by too many connections and bandwidth waste caused by too few connections. Meanwhile, the invention can combine the advantages of the two scheduling methods of the sender driver and the receiver driver to achieve the near-optimal scheduling effect.
To achieve the above object, with reference to fig. 1, the present invention provides a bidirectional coflow scheduling method based on a predicted number of concurrent connections, where the scheduling method includes:
s1, predicting the optimal number of concurrent connections on each host;
s2, collecting information of the mapper and the retarder by adopting a global scheduling program, and generating a global candidate list on the tracker;
s3, determining the number of concurrent connections of each reducer on the local scheduling program according to the dynamic priority list;
s4, readjusting the connection status of each retarder on the micro-scheduler and reporting the current connection status to the global scheduler.
From the foregoing, it can be seen that an important premise for the bidirectional coflow scheduling method of the present invention to be able to perform accurately is that the prediction accuracy of the optimal number of concurrent connections is sufficiently high. Based on this point, in conjunction with fig. 5, the present invention provides a method for predicting the number of optimal concurrent connections suitable for a bidirectional coflow scheduling method, where the prediction method includes:
and adopting a C-SVC model as a multi-class classifier to generate a prediction model to predict the optimal concurrent connection number, wherein the prediction model is used for calculating to obtain the corresponding buffer area size according to the known packet loss rate, and calculating to obtain the current optimal concurrent connection number by combining the relation between the buffer area size and the concurrent connection number.
Preferably, the prediction method further includes:
step one, randomly generating an incast flow containing xi concurrent streamlets, repeating for t times, wherein t is a positive integer larger than 1, obtaining a packet loss rate x corresponding to each concurrent streamlet and a corresponding buffer area size y, and generating a set Q, wherein the set Q contains t groups of different (x, y) value pairs, and the value range of x is [ -1, 1 ].
And step two, regarding the packet loss rate x and the corresponding buffer area size y of a group of concurrent streamlets as a group of sample data, randomly selecting sample data with different buffer area size characteristics from the set Q, and generating a sample data set.
And thirdly, generating a prediction model by using the C-SVC model as a multi-class classifier, and training the prediction model by using the sample data set.
And step four, calculating to obtain the current optimal concurrent connection number by adopting the trained prediction model and combining the relation between the size of the buffer area and the concurrent connection number.
As a preferred example, in step S1, the predicting the optimal number of concurrent connections on each host is to use a support vector machine as a machine learning model to automatically identify the optimal number of concurrent connections, i.e., the queue limit in the switch.
As a preferred example thereof, in step S3, the local scheduler is running on each host for reallocating connections between the sender and the receiver according to a local priority list.
As one preferred example thereof, in step S4, the micro-scheduler is run on each retarder for reacting to connection reallocation from the corresponding local scheduler and adjusting the number of connections per retarder.
As a preferred example, in step S2, the global scheduler runs on the tracker and is used to monitor the network status of each host and dynamically generate the candidate list after receiving the request of the micro scheduler.
As a preferred example, in step S3, the process of determining the number of concurrent connections per decelerator on the local scheduler according to the dynamic priority list includes the following steps:
s31, sorting the speed reducers according to the flow and the residual sizes of the speed reducers by adopting a shortest residual time priority principle;
s32, judging whether the most prior decelerator is compared with the last scheduling sequence and is not changed, if so, entering the step S33, otherwise, entering the step S34;
s33, determining whether the following two conditions are satisfied simultaneously: (1) the idle downlink bandwidth exceeds alpha & b, wherein b is the capacity of a bottleneck link, alpha is a real number parameter between 0 and 1, and (2) the residual connection number is greater than zero, if the residual connection number is simultaneously met, one or more connections are allocated to a suboptimal speed reducer, and the allocated connection number does not exceed half of the idle connection number; otherwise, stopping allocating the connection to the non-preferential reducer, and ending the current flow;
s34, judging whether the residual connection number is less than beta m, wherein m is the predicted connection number, beta is a real number parameter between 0 and 1, if yes, closing the connection of the most prior speed reducer in the last scheduling sequencing; otherwise, when the stream transmission of the most superior reducer in the last scheduling order is completed, the allocated connections are retracted, and β · m connections are allocated to the current most superior reducer, ending the current flow.
As a preferred example, in step S4, the process of readjusting the connection state of each retarder on the micro-scheduler includes the following steps:
s41, judging whether the current connection number of the speed reducer exceeds the distributed connection number, if so, entering a step S42, otherwise, entering a step S43;
s42, adopting a micro-scheduling program to close the connections with the highest flow transmission rate one by one until the current connection number meets the requirement, and ending the current flow;
s43, the micro-scheduler is used to request the candidate list from the tracker to open a connection, connecting the mapper with the largest free uplink capacity if the retarder is the top priority retarder, otherwise connecting the mapper with the largest remaining transmit traffic.
As a preferred example, in step S2, the process of collecting the mapper and retarder information by using the global scheduler and generating the global candidate list on the tracker includes the following steps:
s21, judging whether the speed reducer is the speed reducer with the highest priority, if so, going to step S22, otherwise, going to step S23;
s22, for each sender host, if the cofow to which the reducer belongs also has priority in the sender host, adding a constant lambda.b to the idle uplink bandwidth, wherein lambda is defined as a real number between 0 and 1, and b is defined as the bottleneck link capacity, sorting the sender hosts according to the sequence of the idle uplink bandwidth from small to large, sequentially selecting sorted sender hosts which do not exceed the predicted connection number from the head to generate a global candidate list, and ending the current flow;
s23, if the idle downlink bandwidth of the host to which the reducer belongs is larger than theta.b, theta is a real number parameter between 0 and 1, and b is defined as the capacity of the bottleneck link, sorting the host of the sender according to the sequence of the idle uplink bandwidth from small to large, selecting the idle uplink bandwidth of the host of the sender closest to the idle downlink bandwidth of the host to which the reducer belongs, and sequentially selecting the sorted host of the sender which is sorted in front of the idle uplink bandwidth and does not exceed the predicted connection number from the selected host to generate a global candidate list.
Based on the scheduling method, the invention also provides a bidirectional flow scheduling system based on the predicted concurrent connection number, and the scheduling system comprises:
(1) a prediction model for predicting an optimal number of concurrent connections on each host;
(2) the global scheduler runs on the tracker and is used for collecting the information of the mapper and the retarder and generating a global candidate list on the tracker;
(3) the local scheduling program runs on each host connected with the tracker and is used for determining the number of concurrent connections of each speed reducer connected with the current host according to the dynamic priority list stored in the local host;
(4) and the micro-scheduling program runs on each decelerator and is used for readjusting the connection state of each decelerator and reporting the current connection state to the global scheduling program.
Compared with the prior art, the technical scheme of the invention has the following remarkable beneficial effects:
(1) the invention can automatically identify the optimal number of concurrent connections and combines the advantages of the sender driving and receiver driving scheduling methods.
(2) The accuracy of the present invention estimates the number of concurrent connections to be 98%, reducing the average coflow completion time and the tail coflow completion time by 15% and 40%, respectively.
(3) The invention proposes to use the packet loss rate as a key characteristic for predicting the optimal number of concurrent connections, and establishes a prediction model to predict the optimal number of concurrent connections by measuring the performance of different settings, thereby realizing the accuracy and the high efficiency of the optimal concurrent connection prediction process.
(4) The C-SVC model adopted by the invention consists of [ | Q | (| Q | -1) ]/2 binary classifiers, wherein | Q | is the number of classes. In the training phase, each two-classifier will attempt to partition the specific two classes by solving a convex quadratic program to find a hyperplane. In the test phase, for a given input x, all two classifiers will vote to determine the class number y, where y represents the estimated buffer size. Therefore, the prediction model adopted by the invention can bear estimation errors in a relevant large range, and has stronger robustness.
It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail below can be considered as part of the inventive subject matter of this disclosure unless such concepts are mutually inconsistent. In addition, all combinations of claimed subject matter are considered a part of the presently disclosed subject matter.
The foregoing and other aspects, embodiments and features of the present teachings can be more fully understood from the following description taken in conjunction with the accompanying drawings. Additional aspects of the present invention, such as features and/or advantages of exemplary embodiments, will be apparent from the description which follows, or may be learned by practice of specific embodiments in accordance with the teachings of the present invention.
Drawings
The figures are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. Embodiments of various aspects of the present invention will now be described, by way of example, with reference to the accompanying drawings, in which:
fig. 1 is a flow chart of the bidirectional coflow scheduling method based on the predicted number of concurrent connections of the present invention.
FIG. 2 is a diagram of a bidirectional coflow scheduling architecture.
Fig. 3 is a diagram of an example of a sender-driven non-optimal scheduling method.
Fig. 4 is an exemplary diagram of a receiver-driven non-optimal scheduling method.
Fig. 5 is a flow chart of the optimal number of concurrent connections prediction method of the present invention.
Detailed Description
In order to better understand the technical content of the present invention, specific embodiments are described below with reference to the accompanying drawings.
FIG. 2 is a diagram of a bidirectional coflow scheduling architecture. The local scheduler maintains a dynamic priority list for each reducer on the host. When a new Reducer arrives, the local scheduler will update the priority list, then reallocate the number of connections to each Reducer, and notify the micro-scheduler of the new connection information. After receiving the notification, the micro-scheduler will request the candidate list from the global scheduler, and correspondingly open the new connection or close the old connection according to the connection state generated by the local scheduler. At the same time, the micro-scheduler will report its current connection status to the global scheduler, which is used for information synchronization.
In summary, with reference to fig. 1, the bidirectional flow scheduling method based on the predicted number of concurrent connections in a data center network according to the present invention performs the following steps at each time interval:
1) predicting an optimal number of concurrent connections on each host;
2) collecting the information of mapper and reducer, and generating a global candidate list on the tracker;
3) determining the number of concurrent connections of each reducer on the local scheduler according to the dynamic priority list;
4) the connection state of each reducer is readjusted on the micro-scheduler.
First, the present invention uses a machine learning model to obtain an optimal number of concurrent connections.
Before the predictive model is built, an important experimental result is first explained. For DCTCP, once the aggregation queue length exceeds the size of the switch buffer, the packet loss rate for a group of streams may indicate the size of the switch buffer. The smaller the buffer, the more packets are dropped. Meanwhile, the switch buffer size is a linear function of the ECN marking threshold and the number of concurrent connections. Considering that the packet loss rate is easy to measure and has strong correlation with the size of the buffer, the invention proposes to use the packet loss rate as a key feature for predicting the optimal number of concurrent connections. However, the mathematical relationship between packet loss rate and exchange buffer size is difficult to capture and seems to be non-linear even if all other conditions are fixed. Based on the simplicity and high efficiency of the learning method of the support vector machine, the support vector machine is selected as a learning model, and the C-SVC model is used as a multi-class classifier. Specifically, the invention uses the LibSVM library to construct a machine learning system, and a prediction model is established to predict the optimal number of concurrent connections by measuring the performance of different settings. For example, a C-SVC model is used as a multi-class classifier to generate a prediction model to predict the optimal number of concurrent connections, the prediction model is used for calculating the corresponding buffer area size according to the known packet loss rate, and the current optimal number of concurrent connections is calculated by combining the relation between the buffer area size and the number of concurrent connections.
Specifically, the generation method of the prediction model comprises the following steps:
and S11, repeating the steps t times, and randomly generating an incast flow containing xi concurrent streamlets each time.
Recording the packet loss rate x and the buffer size y of incast flow as a group of data records, and finally obtaining a data set Q containing t groups of records, wherein the range of x is in the range of [ -1, 1], namely the set Q contains t groups of different (x, y) value pairs.
And S12, repeating for several times, randomly selecting a group of packet loss rate x and a buffer area size y from the set Q as a group of sample data each time, and finally obtaining a sample data set containing a certain amount of sample data, wherein the range of x is in the range of [ -1, 1 ].
And S13, dividing the sample data set into a training set and a test set, and using the C-SVC model as a multi-class classifier to obtain a prediction function of the size of the buffer area.
And S14, obtaining the current optimal number of concurrent connections according to the relation between the size of the buffer area and the number of concurrent connections.
It is noted that the C-SVC model consists of [ | Q | (| Q | -1) ]/2 bi-classifiers, where | Q | is the number of classes. In the training phase, each two-classifier will attempt to partition the specific two classes by solving a convex quadratic program to find a hyperplane. In the test phase, for a given input x, all two classifiers will vote to determine the class number y, where y represents the estimated buffer size. The invention can endure estimation error in a relevant large range and has robustness.
Secondly, based on the prediction results, the invention proposes a set of distributed algorithms to perform the flow scheduling.
Fig. 3 and 4 are exemplary diagrams of sender-driven and receiver-driven non-optimal scheduling methods, respectively. The scheduling method of the sender-driven or receiver-driven alone may be sub-optimal when the bandwidth of the host egress link or ingress link becomes a bottleneck. In a real network, these two situations often coexist due to the imbalance of task allocation. The invention provides a group of bidirectional coflow scheduling algorithms, which combines the advantages of a sender driving and a receiver driving scheduling method.
First, a local scheduler is running on each host, responsible for redistributing connections between senders and receivers according to a local priority list, comprising the steps of:
first, applying the Short Remaining Time First (SRTF) principle to sort the reducers according to the residual sizes of the coflow and the reducers.
Secondly, when the most-prior reducer is compared with the last scheduling sequencing and remains unchanged, if the idle downlink bandwidth exceeds alpha & b, wherein b is the capacity of a bottleneck link, alpha is a real number parameter between 0 and 1, and the residual connection number is greater than zero, allocating one or more connections (which cannot exceed half of the idle connection number) to the less-than-optimal reducer; if both conditions are not met, the assignment of the connection to the non-prioritized reducer is stopped.
Thirdly, when the most-prior reducer is changed compared with the last scheduling sequence, if the residual connection number is less than beta · m, wherein m is the predicted connection number, beta is a real number parameter between 0 and 1, closing some connections of the most-prior reducer in the last scheduling sequence; if the number of remaining connections is greater than or equal to β · m, the allocated connection will be reclaimed once the streaming of the most superior reducer in the last scheduling ordering is completed; allocating β · m connections to the current most preferred reducer.
In the above process it is noted that not all connections can be assigned to one reducer, since coflows with priority on the receiver may not have matching priority on the sender. This mismatch also compromises the flow completion time. To solve this problem, the priorities between the sender and the receiver are dynamically balanced using the following steps.
Second, the micro-scheduler runs on each reducer, reacts to connection reallocation from the corresponding local scheduler, and adjusts the number of connections per reducer, including the steps of:
step one, when the current connection number of the reducer exceeds the distributed connection number, the micro-scheduler closes some connections with the highest flow transmission rate one by one until the current connection number meets the requirement;
and secondly, when the number of the connections allocated by the reducer exceeds the current number of the connections, the micro-scheduler requests a candidate list from the tracking scheduler to open more connections, if the reducer is the most prior reducer, the mapper with the maximum free uplink capacity is connected, otherwise, the mapper with the maximum residual sending flow is connected.
The release of the allocated connection in the above steps generally indicates that a preemption is taking place, leaving more bandwidth resources to accelerate the most prioritized reducer. The connection with the highest traffic transmission rate must first be closed. When the number of connections allocated exceeds the current number of connections, the micro-scheduler will request a candidate list from the trace scheduler to open more connections. The candidate list is a subset of the sender hosts of this flow maintained by the tracker, and the algorithm can only select the sender from this list to connect. The following process will discuss the process of generating this candidate list.
Finally, the global scheduler runs on the tracker, monitors the network status of each host, and dynamically generates a candidate list upon receiving the request of the micro scheduler. When the micro-scheduler requests the candidate list, the global scheduler will first query the pending sending master for this retarder. Here, a sending host is said to be in a suspended state if it has not sent any packets to the retarder or has terminated the transmission process due to preemption. The method comprises the following steps:
step one, if the reducer is the most priority, for each sender host, if the flow to which the reducer belongs also has priority in the sender host, adding a constant lambda & b to the idle uplink bandwidth, wherein lambda is defined as a real number between 0 and 1, and b is defined as the bottleneck link capacity (link capacity corresponding to the sender host), sorting the sender hosts from small to large according to the idle uplink bandwidth, and sequentially selecting sorted sender hosts which do not exceed the predicted connection number from the first one to generate a global candidate list;
and secondly, if the reducer is not the most prior, if the idle downlink bandwidth of the host to which the reducer belongs is larger than theta.b, theta is a real number parameter between 0 and 1, and b is defined as the capacity of a bottleneck link (the capacity of a link corresponding to the host of the sender), sorting the hosts of the sender from small to large according to the idle uplink bandwidth, selecting the idle uplink bandwidth of the host of the sender closest to the idle downlink bandwidth of the host to which the reducer belongs, and sequentially selecting the sorted hosts of the sender which are sorted before the idle uplink bandwidth and do not exceed the predicted connection number from the idle uplink bandwidth to generate a global candidate list.
In this disclosure, aspects of the present invention are described with reference to the accompanying drawings, in which a number of illustrative embodiments are shown. Embodiments of the present disclosure are not necessarily defined to include all aspects of the invention. It should be appreciated that the various concepts and embodiments described above, as well as those described in greater detail below, may be implemented in any of numerous ways, as the disclosed concepts and embodiments are not limited to any one implementation. In addition, some aspects of the present disclosure may be used alone, or in any suitable combination with other aspects of the present disclosure.
Although the present invention has been described with reference to the preferred embodiments, it is not intended to be limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims (1)

1. A bi-directional coflow scheduling system based on a predicted number of concurrent connections, the scheduling system comprising:
a prediction model for predicting an optimal number of concurrent connections on each host;
the global scheduler runs on the tracker and is used for collecting the information of the mapper and the retarder and generating a global candidate list on the tracker;
the local scheduling program runs on each host connected with the tracker and is used for determining the number of concurrent connections of each speed reducer connected with the current host according to the dynamic priority list stored in the local host;
the micro-scheduling program runs on each speed reducer, is used for readjusting the connection state of each speed reducer and reports the current connection state to the global scheduling program; the bidirectional coflow scheduling method based on the predicted concurrent connection number applied by the scheduling system comprises the following steps:
s1, predicting the optimal number of concurrent connections on each host;
s2, collecting information of the mapper and the retarder by adopting a global scheduling program, and generating a global candidate list on the tracker;
s3, determining the number of concurrent connections of each reducer on the local scheduling program according to the dynamic priority list;
s4, readjusting the connection state of each reducer on the micro-scheduling program, and reporting the current connection state to the global scheduling program;
in step S1, the predicting the optimal number of concurrent connections on each host refers to automatically identifying the optimal number of concurrent connections, that is, the queue limit in the switch, by using a support vector machine as a machine learning model;
the prediction method comprises the following steps: the method comprises the steps that a C-SVC model is used as a multi-class classifier, a prediction model is generated to predict the optimal concurrent connection number, the prediction model is used for calculating to obtain the size of a corresponding buffer area according to the known packet loss rate, and the current optimal concurrent connection number is calculated by combining the size of the buffer area and the relationship of the concurrent connection number;
step one, randomly generating an incast flow containing xi concurrent streamlets, repeating for t times, wherein t is a positive integer larger than 1, acquiring a packet loss rate x corresponding to each concurrent streamlet and a corresponding buffer area size y, and generating a set Q, wherein the set Q contains t groups of different (x, y) value pairs, and the value range of x is [ -1, 1 ];
step two, regarding the packet loss rate x and the corresponding buffer area size y of a group of concurrent streamlets as a group of sample data, randomly selecting sample data with different buffer area size characteristics from the set Q, and generating a sample data set;
step three, generating a prediction model by using a C-SVC model as a multi-class classifier, and training the prediction model by using a sample data set;
step four, calculating to obtain the current optimal number of concurrent connections by adopting a trained prediction model and combining the relation between the size of the buffer area and the number of concurrent connections;
in step S3, the local scheduler is running on each host for redistributing connections between the sender and the receiver according to the local priority list;
in step S4, the micro-scheduler runs on each reducer, and is configured to respond to connection reallocation from a corresponding local scheduler, and adjust the number of connections for each reducer;
in step S2, the global scheduler runs on the tracker and is configured to monitor a network status of each host and dynamically generate a candidate list after receiving a request from the micro scheduler;
in step S3, the process of determining the number of concurrent connections of each retarder on the local scheduler according to the dynamic priority list includes the following steps:
s31, sorting the speed reducers according to the flow and the residual sizes of the speed reducers by adopting a shortest residual time priority principle;
s32, judging whether the most prior decelerator is compared with the last scheduling sequence and is not changed, if so, entering the step S33, otherwise, entering the step S34;
s33, determining whether the following two conditions are satisfied at the same time: (1) the idle downlink bandwidth exceeds alpha & b, wherein b is the capacity of a bottleneck link, alpha is a real number parameter between 0 and 1, and (2) the residual connection number is greater than zero, if the residual connection number is simultaneously met, one or more connections are allocated to a suboptimal speed reducer, and the allocated connection number does not exceed half of the idle connection number; otherwise, stopping allocating the connection to the non-preferential reducer, and ending the current flow;
s34, judging whether the residual connection number is less than beta m, wherein m is the predicted connection number, beta is a real number parameter between 0 and 1, if yes, closing the connection of the most prior speed reducer in the last scheduling sequencing; otherwise, when the stream transmission of the most-prior reducer in the last scheduling sequence is finished, the distributed connections are recovered, the beta m connections are distributed to the current most-prior reducer, and the current process is finished;
in step S4, the process of readjusting the connection state of each retarder on the micro-scheduler includes the steps of:
s41, judging whether the current connection number of the speed reducer exceeds the distributed connection number, if so, entering a step S42, otherwise, entering a step S43;
s42, adopting a micro-scheduling program to close the connections with the highest flow transmission rate one by one until the current connection number meets the requirement, and ending the current flow;
s43, using micro-scheduling program to request candidate list to tracker to open connection, if reducer is the most advanced reducer, connecting mapper with maximum free uplink capacity, otherwise connecting mapper with maximum residual sending flow;
in step S2, the process of collecting the mapper and retarder information by using the global scheduler and generating a global candidate list on the tracker includes the following steps:
s21, judging whether the speed reducer is the speed reducer with the highest priority, if so, going to step S22, otherwise, going to step S23;
s22, for each sender host, if the flow to which the reducer belongs also has priority in the sender host, adding a constant lambda & b to the idle uplink bandwidth, wherein lambda is defined as a real number between 0 and 1, and b is defined as the bottleneck link capacity, sorting the sender hosts according to the order of the idle uplink bandwidth from small to large, sequentially selecting sorted sender hosts which do not exceed the predicted connection number from the head to generate a global candidate list, and ending the current flow;
s23, if the idle downlink bandwidth of the host to which the reducer belongs is larger than theta.b, theta is a real number parameter between 0 and 1, and b is defined as the capacity of the bottleneck link, sorting the host of the sender according to the sequence of the idle uplink bandwidth from small to large, selecting the idle uplink bandwidth of the host of the sender closest to the idle downlink bandwidth of the host to which the reducer belongs, and sequentially selecting the sorted host of the sender which is sorted in front of the idle uplink bandwidth and does not exceed the predicted connection number from the selected host to generate a global candidate list.
CN202010360227.2A 2020-04-30 2020-04-30 Bidirectional coflow scheduling system based on prediction of concurrent connection number Active CN111585904B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010360227.2A CN111585904B (en) 2020-04-30 2020-04-30 Bidirectional coflow scheduling system based on prediction of concurrent connection number

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010360227.2A CN111585904B (en) 2020-04-30 2020-04-30 Bidirectional coflow scheduling system based on prediction of concurrent connection number

Publications (2)

Publication Number Publication Date
CN111585904A CN111585904A (en) 2020-08-25
CN111585904B true CN111585904B (en) 2022-07-08

Family

ID=72111942

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010360227.2A Active CN111585904B (en) 2020-04-30 2020-04-30 Bidirectional coflow scheduling system based on prediction of concurrent connection number

Country Status (1)

Country Link
CN (1) CN111585904B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107580052A (en) * 2017-09-07 2018-01-12 翼果(深圳)科技有限公司 From the network self-adapting reptile method and system of evolution
CN108809764A (en) * 2018-06-15 2018-11-13 东南大学 The network based on GPU accesses overtrick and connects number evaluation method under sliding window
CN110990140A (en) * 2019-12-10 2020-04-10 电子科技大学 Method for scheduling distributed machine learning flow in photoelectric switching network

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105227488B (en) * 2015-08-25 2018-05-08 上海交通大学 A kind of network flow group scheduling method for distributed computer platforms

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107580052A (en) * 2017-09-07 2018-01-12 翼果(深圳)科技有限公司 From the network self-adapting reptile method and system of evolution
CN108809764A (en) * 2018-06-15 2018-11-13 东南大学 The network based on GPU accesses overtrick and connects number evaluation method under sliding window
CN110990140A (en) * 2019-12-10 2020-04-10 电子科技大学 Method for scheduling distributed machine learning flow in photoelectric switching network

Also Published As

Publication number Publication date
CN111585904A (en) 2020-08-25

Similar Documents

Publication Publication Date Title
CN107770096B (en) SDN/NFV network dynamic resource allocation method based on load balancing
CN106789660B (en) QoS sensible flow management method in software defined network
US6680906B1 (en) Regulating packet traffic in an integrated services network
US7027457B1 (en) Method and apparatus for providing differentiated Quality-of-Service guarantees in scalable packet switches
US20220045972A1 (en) Flow-based management of shared buffer resources
KR100748187B1 (en) Node availability prediction-based grid network congestion control device and method therefor
JP3306705B2 (en) Packet transfer control device and scheduling method thereof
US9559956B2 (en) Sharing bandwidth among multiple users of network applications
RU2005131960A (en) ACCESS PERMISSION MANAGEMENT AND RESOURCE ALLOCATION IN THE COMMUNICATION SYSTEM WITH SUPPORT OF APPLICATION STREAMS WITH AVAILABILITY OF SERVICE QUALITY REQUIREMENTS
CN113141590B (en) Industrial Internet of things-oriented wireless communication scheduling method and device
CN101692648B (en) Method and system for queue scheduling
CN104579962A (en) Method and device for differentiating QoS strategies of different messages
CN104092625B (en) A kind of self adaptation being used in DCN asks dispatching method in batches
US7843825B2 (en) Method and system for packet rate shaping
Li et al. OPTAS: Decentralized flow monitoring and scheduling for tiny tasks
EP1528728A1 (en) Packet scheduling based on quality of service and index of dispersion for counts
CN103634223A (en) Network service flow based dynamic control transmission method and device
CN1518296A (en) Method of implementing integrated queue scheduling for supporting multi service
CN111585904B (en) Bidirectional coflow scheduling system based on prediction of concurrent connection number
CN102469602A (en) Method for user multi-service dispatching
CN111585905B (en) Optimal concurrent connection number prediction method suitable for bidirectional coflow scheduling method
CN107404454B (en) Call quality adjusting method and device
CN114390043A (en) Method for achieving multi-path QUIC transmission optimization based on enterprise-level cloud storage system
CN104066138A (en) Group switching method based on service priority in blocking rate constraint
US20230022037A1 (en) Flow-based management of shared buffer resources

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant