CN116319378B - Network traffic matrix estimation and model training method and system based on deep learning - Google Patents
Network traffic matrix estimation and model training method and system based on deep learning Download PDFInfo
- Publication number
- CN116319378B CN116319378B CN202310545574.6A CN202310545574A CN116319378B CN 116319378 B CN116319378 B CN 116319378B CN 202310545574 A CN202310545574 A CN 202310545574A CN 116319378 B CN116319378 B CN 116319378B
- Authority
- CN
- China
- Prior art keywords
- flow
- source point
- network
- matrix
- deep learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000011159 matrix material Substances 0.000 title claims abstract description 94
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012549 training Methods 0.000 title claims abstract description 39
- 238000013135 deep learning Methods 0.000 title claims abstract description 33
- 238000013507 mapping Methods 0.000 claims abstract description 30
- 230000006870 function Effects 0.000 claims description 44
- 230000004913 activation Effects 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 11
- 230000003213 activating effect Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000003062 neural network model Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 230000007246 mechanism Effects 0.000 abstract description 2
- 239000000284 extract Substances 0.000 abstract 1
- 238000012360 testing method Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 5
- ZEFNOZRLAWVAQF-UHFFFAOYSA-N Dinitolmide Chemical compound CC1=C(C(N)=O)C=C([N+]([O-])=O)C=C1[N+]([O-])=O ZEFNOZRLAWVAQF-UHFFFAOYSA-N 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000000354 decomposition reaction Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 238000005111 flow chemistry technique Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 238000003325 tomography Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
Landscapes
- Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Artificial Intelligence (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention relates to the technical field of network engineering and artificial intelligence, in particular to a method and a system for estimating a network flow matrix and training a model based on deep learning. The invention trains aiming at a basic model comprising a mapping module and a reconstruction module, and extracts the mapping module after training as a network flow matrix estimation model. The mapping module calculates a flow matrix in combination with the link load, and the reconstruction module reconstructs a link load estimated value in combination with the flow matrix. According to the method, under the condition that the data set is incomplete, accuracy, robustness and instantaneity of real TM estimation by the basic model can be guaranteed by using a double-loss mechanism, and the problem of model under-convergence caused by missing of a large number of training data sets in actual life is solved.
Description
Technical Field
The invention relates to the technical field of network engineering and artificial intelligence, in particular to a method and a system for estimating a network flow matrix and training a model based on deep learning.
Background
Traffic Matrix (TM) is used to represent the Traffic demand size between all possible network nodes, which is often referred to as source-flow (OD-flows). The network state reflected by the TM is of great significance in many network management problems, such as traffic engineering, anomaly detection, and capacity planning. However, as the network scale continues to expand, it has become impractical to measure the source traffic directly by collecting each packet delivery trace. Currently, a more feasible solution is to solve the system of underdetermined equations by using network tomography (Network Tomography, NT for short), and estimate TM at the corresponding moment by Link-level traffic (Link-loads).
More and more scholars have been engaged in the field of network traffic estimation in the last decade, and the main current TM estimation methods can be divided into three categories: NT-based schemes, matrix decomposition-based schemes and deep learning-based schemes. In the first category of schemes, the distribution of TM is assumed to be estimated as accurately as possible, classical poisson distribution by Vardi et al and gravity model by Zhang et al. Such an estimation scheme is extremely dependent on the preconditions established. The second category of schemes attempts to recover missing traffic sizes using Singular Value Decomposition (SVD) and Principal Component Analysis (PCA). For example, roughan et al propose a sparse regularized matrix decomposition (SRMF) method. The two schemes rely on hypothesis and statistical techniques, and have large differences in accuracy, so that the requirement of accurate estimation is difficult to meet.
The third category of deep learning-based schemes learns the correspondence between link load and source point traffic by means of a deep neural network, and then estimates TM using the trained model with the link load sequence as input. Unlike the two approaches above, this approach based on deep learning does not rely on any assumptions and statistical techniques while ensuring higher accuracy. Jiang et al first introduced a Back Propagation Neural Network (BPNN) to estimate the network traffic matrix. Next Nie et al also constructed a Deep Belief Network (DBN) based flow estimation model. Then Convolutional Neural Network (CNN), cyclic neural network (LSTM) and variational self-encoder (VAE) are also put into the task of TM estimation.
While deep learning based methods can achieve ideal results with minimal assumptions, almost all methods require sufficient and complete flow data to support training of the model. In other words, considering that the situation that the loss and the error can almost occur in the process of measuring the source flow, no TM estimation scheme conforming to the actual situation has been proposed.
Disclosure of Invention
In order to overcome the defect that in the prior art, the application of a deep learning model to TM estimation is difficult to achieve an ideal effect due to source point flow loss, the invention provides a training method of a network flow matrix estimation model based on deep learning.
The invention provides a network traffic matrix estimation model training method based on deep learning, which comprises the following steps:
s1, constructing a basic model, wherein the basic model comprises a mapping module and a reconstruction module; the mapping module and the reconstruction module are both neural network models;
let the set of source point pairs of known traffic in the network be Q, and the set of source point pairs of unknown traffic be U;
the input of the mapping module is a link load; the output of the mapping module is a source point flow estimation matrix X1 of the network, and the X1 is used for describing flow calculation values of each source point pair in the network;
the input of the reconstruction module is a source point flow combination matrix X, wherein X is used for describing the flow of each source point pair in the network, and X is formed by combining the actual flow of the source point pair of the known flow and the flow calculation value of the source point pair of the unknown flow; the output of the reconstruction module is a link load estimated value Y' corresponding to the source point flow combination matrix X;
s2, acquiring a source point flow observation matrix X0 marked with a link load true value Y as a learning sample, wherein an observed value of a source point pair of known flow in the source point flow observation matrix X0 is an actual flow; calculating a first loss function according to the difference between the actual flow of the source point pair of the known flow and the flow calculated value, calculating a second loss function according to the difference between the link load estimated value Y' and the link load true value Y, and constructing a total loss function according to the first loss function and the second loss function;
and S3, enabling the basic model to learn the selected learning sample, and reversely updating the basic model by combining the total loss function until the basic model reaches a set convergence condition, and extracting the mapping network as a network flow matrix estimation model and outputting the network flow matrix estimation model.
Preferably, in the source point flow observation matrix X0, the observed value of the source point pair of the unknown flow is the average value of the actual flow of the source point pair of the known flow.
Preferably, the first loss function is the mean square error between the actual flow and the calculated flow value of the source point pair of the known flow; the second loss function is the mean square error between the link load estimate Y' and the link load true Y.
Preferably, the total loss function is the sum of the first loss function and the second loss function.
Preferably, the convergence condition is: the number of iterations reaches a set threshold, or the total loss reaches a set loss threshold.
Preferably, the mapping module comprises two nonlinear units, a flattening unit and a Sigmoid activating unit which are sequentially connected, wherein the input of the first nonlinear unit is used as the input of the mapping module, and the output of the Sigmoid activating unit is the output of the mapping module; the nonlinear unit is used for carrying out nonlinear processing on the input, the flattening unit maps the input to a space with the same dimension as the source point flow matrix, and the output of the flattening unit is activated by the Sigmoid activation unit and then is output as the source point flow estimation matrix.
Preferably, the nonlinear unit comprises a linear fully-connected layer and a ReLu function activation layer; the input of the linear full-connection layer in the first nonlinear unit is the input of the mapping module; the linear full-connection layer of the first nonlinear unit, the ReLu function activation layer of the first nonlinear unit, the linear full-connection layer of the second nonlinear unit, the ReLu function activation layer of the second nonlinear unit, the flattening unit and the Sigmoid activation unit are sequentially connected.
The invention also provides a network flow matrix estimation method based on deep learning, which adopts the network flow matrix estimation model to realize the flow matrix estimation of any network, and comprises the following steps:
SA1, acquiring historical flow data of a network to be estimated, acquiring actual flow of a source point pair of known flow at the same moment from the historical flow data to construct a source point flow observation matrix X0, and acquiring a link load true value Y at the moment corresponding to X0 to construct a learning sample (X0, Y);
SA2, acquiring a network flow matrix estimation model by adopting the network flow matrix estimation model training method based on deep learning;
and SA3, inputting the link load of the network to be estimated into a network flow matrix estimation model, and obtaining a source point flow estimation matrix X1 output by the network flow matrix estimation model.
The invention also provides a network traffic matrix estimation system based on deep learning, which comprises a memory, wherein the memory stores a computer program, and the computer program realizes the network traffic matrix estimation method based on deep learning when being executed.
Preferably, the system further comprises a processor, wherein the processor is connected with the memory, and the processor is used for executing the computer program to realize the network traffic matrix estimation method based on deep learning.
The invention has the advantages that:
(1) According to the training method of the network traffic matrix estimation model based on deep learning, provided by the invention, under the condition that a data set is incomplete, accuracy, robustness and instantaneity of real TM estimation of a basic model can be ensured by utilizing a double-loss mechanism, and the problem of model undercrown caused by a large number of training data sets in actual life is solved.
(2) After the network flow matrix estimation model provided by the invention is trained on the data set containing various unknown ratios, the accuracy and precision are obviously improved compared with the prior scheme no matter the estimation is carried out on the source point pair of the known flow or the source point pair of the unknown flow.
(3) The network flow matrix estimation model provided by the invention adopts a lightweight deep neural network, and has the advantages of high interpretability, short estimation time, low model redundancy and the like, so that the network real-time estimation applicability of the invention is improved.
(4) The method calculates the second loss function aiming at the reconstruction module, forces the depth model to continuously optimize the estimation of the unknown flow in the training process, enables the network flow matrix estimation model obtained by training to display stronger unknown flow processing capacity, and can reduce the performance loss as far as possible when the unknown flow is lost, thereby remotely leading the estimation accuracy of the unknown flow.
Drawings
FIG. 1 is a flow chart of a training method of a network traffic matrix estimation model based on deep learning.
Fig. 2 is a schematic diagram of a base model and a loss function.
FIG. 3 is a comparison of the accuracy of different models on the data set Abilene;
fig. 4 is a comparison of the accuracy of different models on the dataset GEANT.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the training method of the network traffic matrix estimation model based on deep learning in this embodiment is used for training the network traffic matrix estimation model.
Let the set of source point pairs of known traffic in the network be Q and the set of source point pairs of unknown traffic be U.
Let (i, j) represent the combination of the i-th source point and the j-th source point in the network; if there is a channel between the ith and jth source points, (i, j) forms a source point pair.
The source point traffic combination matrix X is used to describe the traffic of each source point pair in the network:
X={a(i,j)|1≦i≦n,1≦j≦n}
the ith source point and the jth source point form a source point pair, and a (i, j) is the source point flow between the ith source point and the jth source point; a (i, j) =0 if there is no channel between the i-th source point and the j-th source point.
In this embodiment, a source point traffic observation matrix X0 is defined;
X0={a0(i,j)|1≦i≦n,1≦j≦n}
when (i, j) is not a source point pair, then a0 (i, j) =0;
when (i, j) e Q, then a0 (i, j) is the measurement;
when (i, j) ∈u, a0 (i, j) is calculated according to the set function.
In the present embodiment, there are defined:
the number of source point pairs contained in Q is m, and the sum of the actual flow of the source point pairs of known flow is A (Q);
A(Q)=∑ (i,j)∈Q [a0(i,j)]
when (i, j) ∈u, a0 (i, j) =a (Q)/m.
Let X (0, Q) = { a0 (i, j) | (i, j) ∈q }, X (0,U) = { a0 (i, j) | (i, j) ∈u }, then:
X0=X(0,Q)||X(0,U)||{a0(i,j)=0|(i,j)∈R}；
r represents a set of source point combinations in the network without channels between each other; the i represents the data space concatenation.
X (0, Q) represents the actual flow set of source point pairs for known flows, and X (0,U) represents the flow observation set of source point pairs for unknown flows;
in the present embodiment, the following steps S1 to S3 are adopted in training the network traffic matrix estimation model.
S1, constructing a basic model, wherein the basic model comprises a mapping module and a reconstruction module; the mapping module and the reconstruction module are both neural network models;
the input of the mapping module is a link load; the output of the mapping module is a source point flow estimation matrix X1 of the network, and the X1 is used for describing flow calculation values of each source point pair in the network;
let a1 (i, j) represent the flow calculation value between the ith and jth source points, X (1, q) represent the flow calculation value set of the source point pair of the known flow, X (1, u) represent the flow calculation value set of the source point pair of the unknown flow, namely:
X(1,Q)={a1(i,j)|(i,j)∈Q}
X(1,U)={a1(i,j)|(i,j)∈U}
then: x1=x (1, q) ||x (1, u) |a 0 (i, j) =0| (i, j) ∈r).
In this embodiment, the mapping module includes: two nonlinear units, a flattening unit and a Sigmoid activating unit; the nonlinear unit comprises a linear full-connection layer and a ReLu function activation layer. The input of the linear full-connection layer in the first nonlinear unit is the input of the mapping module, and the output of the Sigmoid activating unit is the output of the mapping module.
The linear full-connection layer of the first nonlinear unit, the ReLu function activation layer of the first nonlinear unit, the linear full-connection layer of the second nonlinear unit, the ReLu function activation layer of the second nonlinear unit, the flattening unit and the Sigmoid activation unit are sequentially connected.
The nonlinear unit is used for carrying out nonlinear processing on the input, the flattening unit maps the input to a space with the same dimension as the flow matrix, and the output of the flattening unit is activated by the Sigmoid activation unit and then is output as the flow matrix.
The input of the reconstruction module is a source point flow combination matrix X, wherein X is used for describing the flow of each source point pair in the network, and X is formed by combining the actual flow of the source point pair of the known flow and the flow calculation value of the source point pair of the unknown flow;
X==X(0,Q)||X(1,U)||{a0(i,j)=0|(i,j)∈R}。
the output of the reconstruction module is a link load estimated value Y' corresponding to the source point flow combination matrix X.
S2, acquiring a source point flow observation matrix X0 marked with a link load true value Y as a learning sample, calculating a first loss function by combining the difference value between the actual flow and the flow calculation value of a source point pair of known flow, calculating a second loss function by combining the difference value between the link load estimation value Y' and the link load true value Y, and constructing a total loss function by combining the first loss function and the second loss function.
In this embodiment, the first loss function is the Mean Square Error (MSE) between the actual flow and the calculated flow value of the source point pair of the known flow, i.e., the mean square error of X (0, q) and X (1, q); the second loss function is the mean square error between the link load estimate Y' and the link load true Y, and the total loss function is the sum of the first and second loss functions.
And S3, enabling the basic model to learn the selected learning sample, and reversely updating the basic model by combining the total loss function until the basic model reaches a set convergence condition, and extracting the mapping network as a network flow matrix estimation model and outputting the network flow matrix estimation model.
According to the network flow matrix estimation method based on deep learning, firstly, a learning sample is built by combining historical flow data of a network to be estimated, the network flow matrix estimation model is trained by combining the built learning sample by adopting the network flow matrix estimation model training method based on deep learning, and then, the network flow matrix estimation model can directly output a source point flow estimation matrix X1 corresponding to the network to be estimated only by inputting a link load of the network to be estimated into the obtained network flow matrix estimation model.
The network traffic matrix estimation model provided by the invention is verified by combining a specific embodiment.
In this embodiment, in order to verify the performance of the flow matrix estimation model provided by the present invention, the network flow matrix estimation model Auto Tomo provided by the present invention is compared with three other existing deep learning models under various unknown flow ratios by combining different data sets.
In this embodiment two real world data sets Abilene and G É ANT are used, respectively. Where the network measured by the abile dataset contains 12 routes, 54 directed links. This dataset collected all OD-flows in the network at 5 minute intervals during 3 to 9 months 2004. Whereas there are 23 routes and 120 directional links in the G É ANT network, the data set was acquired at 15 minute intervals during months 1 to 4 of 2004.
The three existing models selected in this embodiment are a variational self-encoder (VAE) estimation model, a Deep Belief Network (DBN) estimation model, and a Back Propagation Neural Network (BPNN) based model MNETME, respectively.
The VAE first learns the true distribution of TM (traffic matrix) and then uses gradient descent to find solutions that satisfy both the training set distribution and the tomogram equation. And the DBN and the MNETME train respective depth models, and the Link-loads (Link load) sequences directly output TM estimates meeting the conditions.
In this embodiment, the source traffic in the data set is known, and the method for verifying the different models for the data set is as follows: firstly, dividing a data set into a preparation training set and a test set; then, partial data in the preliminary training set is replaced with unknown flow according to different proportions, and the replaced preliminary training set is used as a training set; and respectively training each model by combining the training set and the testing set.
In this embodiment, the normalized average absolute error is used as an accuracy evaluation index of the model.
FIG. 3 shows the normalized mean absolute error achieved by the obtained model at the time of testing, using VAE, DBN, MNETME and Auto Tomo respectively in combination with the dataset Abilene for learning.
FIG. 4 shows the normalized mean absolute error achieved by the obtained model at the time of testing, as learned by DBN, MNETME and Auto-Tomo in combination with dataset G É ANT, respectively.
In fig. 3 and 4, the unknown traffic ratio is the ratio of the source traffic that is replaced with unknown in the preliminary training set in the traffic matrix.
As can be seen in connection with fig. 3 and 4, the accuracy of all models is inversely proportional to the unknown ratio, especially as reflected on the high-dimensional dataset gent. The accuracy of the network flow estimation model in the invention is better than other depth models under any unknown proportion of any data set. The higher the unknown ratio is, the larger the accuracy difference of the comparison model relative to the network flow estimation model in the invention is, so that the network flow estimation model provided by the invention has stronger robustness and the performance under the condition of source point flow loss is far superior to the prior art. The method is far ahead in the estimation accuracy of the unknown flow, shows stronger unknown flow processing capacity, and can reduce performance loss as much as possible when the unknown flow is lost.
The time duration used for each model is shown in table 1 for the estimated time. As can be seen from table 1, the training time of the network traffic matrix estimation model provided by the invention on the low latitude data set Abilene and the high latitude data set GEANT is rapidly reduced, the testing time has a small duty ratio relative to the training time, the lightweight model and the simple mapping process can be derived from the testing time, and the speed advantage of the network traffic matrix estimation model provided by the invention becomes more obvious along with the improvement of the dimensionality of the data set in the training stage. Therefore, with the continuous expansion of the current network scale, the invention better fits the development trend and the real-time application of the large-scale network.
Table 1 training time and test time of four models on two real data sets
The above embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention, and any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. The network traffic matrix estimation model training method based on deep learning is characterized by comprising the following steps of:
s1, constructing a basic model, wherein the basic model comprises a mapping module and a reconstruction module; the mapping module and the reconstruction module are both neural network models;
let the set of source point pairs of known traffic in the network be Q, and the set of source point pairs of unknown traffic be U;
the input of the mapping module is a link load; the output of the mapping module is a source point flow estimation matrix X1 of the network, and the X1 is used for describing flow calculation values of each source point pair in the network;
the input of the reconstruction module is a source point flow combination matrix X, wherein X is used for describing the flow of each source point pair in the network, and X is formed by combining the actual flow of the source point pair of the known flow and the flow calculation value of the source point pair of the unknown flow; the output of the reconstruction module is a link load estimated value Y' corresponding to the source point flow combination matrix X;
s2, acquiring a source point flow observation matrix X0 marked with a link load true value Y as a learning sample, wherein an observed value of a source point pair of known flow in the source point flow observation matrix X0 is an actual flow; calculating a first loss function according to the difference between the actual flow of the source point pair of the known flow and the flow calculated value, calculating a second loss function according to the difference between the link load estimated value Y' and the link load true value Y, and constructing a total loss function according to the first loss function and the second loss function;
and S3, enabling the basic model to learn the selected learning sample, and reversely updating the basic model by combining the total loss function until the basic model reaches a set convergence condition, and extracting the mapping network as a network flow matrix estimation model and outputting the network flow matrix estimation model.
2. The training method of network traffic matrix estimation model based on deep learning as claimed in claim 1, wherein in the source traffic observation matrix X0, the observed value of the source point pair of unknown traffic is the average value of the actual traffic of the source point pair of known traffic.
3. The training method of network traffic matrix estimation model based on deep learning as claimed in claim 1, wherein the first loss function is the mean square error between the actual traffic of the source point pair of the known traffic and the traffic calculation value; the second loss function is the mean square error between the link load estimate Y' and the link load true Y.
4. The deep learning based network traffic matrix estimation model training method of claim 1, wherein the total loss function is a sum of the first loss function and the second loss function.
5. The training method of network traffic matrix estimation model based on deep learning as claimed in claim 1, wherein the convergence condition is: the number of iterations reaches a set threshold, or the total loss reaches a set loss threshold.
6. The training method of network traffic matrix estimation model based on deep learning as claimed in claim 1, wherein the mapping module comprises two nonlinear units, a flattening unit and a Sigmoid activating unit connected in sequence, the input of the first nonlinear unit is used as the input of the mapping module, and the output of the Sigmoid activating unit is the output of the mapping module; the nonlinear unit is used for carrying out nonlinear processing on the input, the flattening unit maps the input to a space with the same dimension as the source point flow matrix, and the output of the flattening unit is activated by the Sigmoid activation unit and then is output as the source point flow estimation matrix.
7. The deep learning-based network traffic matrix estimation model training method of claim 6, wherein the nonlinear unit comprises a linear full connection layer and a ReLu function activation layer; the input of the linear full-connection layer in the first nonlinear unit is the input of the mapping module; the linear full-connection layer of the first nonlinear unit, the ReLu function activation layer of the first nonlinear unit, the linear full-connection layer of the second nonlinear unit, the ReLu function activation layer of the second nonlinear unit, the flattening unit and the Sigmoid activation unit are sequentially connected.
8. The network traffic matrix estimation method based on deep learning is characterized by comprising the following steps of:
SA1, acquiring historical flow data of a network to be estimated, acquiring actual flow of a source point pair of known flow at the same moment from the historical flow data to construct a source point flow observation matrix X0, and acquiring a link load true value Y at the moment corresponding to X0 to construct a learning sample (X0, Y);
SA2, acquiring a network traffic matrix estimation model by adopting the training method of the network traffic matrix estimation model based on the deep learning as claimed in any one of claims 1 to 7;
and SA3, inputting the link load of the network to be estimated into a network flow matrix estimation model, and obtaining a source point flow estimation matrix X1 output by the network flow matrix estimation model.
9. A deep learning based network traffic matrix estimation system comprising a memory storing a computer program that when executed implements the deep learning based network traffic matrix estimation method of claim 8.
10. The deep learning based network traffic matrix estimation system of claim 9, further comprising a processor coupled to the memory, the processor configured to execute the computer program to implement the deep learning based network traffic matrix estimation method of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310545574.6A CN116319378B (en) | 2023-05-16 | 2023-05-16 | Network traffic matrix estimation and model training method and system based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310545574.6A CN116319378B (en) | 2023-05-16 | 2023-05-16 | Network traffic matrix estimation and model training method and system based on deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116319378A CN116319378A (en) | 2023-06-23 |
| CN116319378B true CN116319378B (en) | 2023-07-21 |
Family
ID=86789045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310545574.6A Active CN116319378B (en) | 2023-05-16 | 2023-05-16 | Network traffic matrix estimation and model training method and system based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116319378B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118101497B (en) * | 2024-04-23 | 2024-07-02 | 合肥工业大学 | Network traffic matrix estimation model, method and system based on generated flow model |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018207241A (en) * | 2017-05-31 | 2018-12-27 | 富士通株式会社 | Management device, management method, and management program |
| CN110851782A (en) * | 2019-11-12 | 2020-02-28 | 南京邮电大学 | Network flow prediction method based on lightweight spatiotemporal deep learning model |
| WO2020253690A1 (en) * | 2019-06-17 | 2020-12-24 | 浙江大学 | Deep learning beam domain channel estimation method based on approximate message passing algorithm |
| WO2021186158A1 (en) * | 2020-03-17 | 2021-09-23 | The University Court Of The University Of Edinburgh | A distributed network traffic data decomposition method |
| CN115022193A (en) * | 2022-05-23 | 2022-09-06 | 电子科技大学 | Local area network flow prediction method based on improved ConvLSTM deep learning model |
| CN116094758A (en) * | 2022-12-02 | 2023-05-09 | 肇庆医学高等专科学校 | Large-scale network flow acquisition method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11443166B2 (en) * | 2018-10-29 | 2022-09-13 | Oracle International Corporation | Datacenter level utilization prediction without operating system involvement |
-
2023
- 2023-05-16 CN CN202310545574.6A patent/CN116319378B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018207241A (en) * | 2017-05-31 | 2018-12-27 | 富士通株式会社 | Management device, management method, and management program |
| WO2020253690A1 (en) * | 2019-06-17 | 2020-12-24 | 浙江大学 | Deep learning beam domain channel estimation method based on approximate message passing algorithm |
| CN110851782A (en) * | 2019-11-12 | 2020-02-28 | 南京邮电大学 | Network flow prediction method based on lightweight spatiotemporal deep learning model |
| WO2021186158A1 (en) * | 2020-03-17 | 2021-09-23 | The University Court Of The University Of Edinburgh | A distributed network traffic data decomposition method |
| CN115022193A (en) * | 2022-05-23 | 2022-09-06 | 电子科技大学 | Local area network flow prediction method based on improved ConvLSTM deep learning model |
| CN116094758A (en) * | 2022-12-02 | 2023-05-09 | 肇庆医学高等专科学校 | Large-scale network flow acquisition method and system |
Non-Patent Citations (2)
| Title |
|---|
| 基于局域相关向量机回归模型的小尺度网络流量的非线性预测;孟庆芳;陈月辉;冯志全;王枫林;陈珊珊;;物理学报(15);全文 * |
| 基于数据中心流量特征的端到端流量估计算法;乔焰，焦俊，饶元;计算机科学;第44卷(第2期);171-175 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116319378A (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108960303B (en) | Unmanned aerial vehicle flight data anomaly detection method based on LSTM | |
| CN110427654B (en) | Landslide prediction model construction method and system based on sensitive state | |
| CN112115419B (en) | System state estimation method and system state estimation device | |
| CN103399281B (en) | Based on the ND-AR model of cycle life deterioration stage parameter and the cycle life of lithium ion battery Forecasting Methodology of EKF method | |
| CN110674752A (en) | Hidden Markov model-based tool wear state identification and prediction method | |
| CN116319378B (en) | Network traffic matrix estimation and model training method and system based on deep learning | |
| CN106707221B (en) | Sensor fault diagnosis method and system in a kind of electric energy metering device | |
| WO2022021726A1 (en) | Pmu-based power system state estimation performance evaluation method | |
| CN105335619B (en) | A kind of collaboration optimization of process of rock-burst mathematical calculation model parameter back-analysis | |
| CN111191726B (en) | Fault classification method based on weak supervision learning multilayer perceptron | |
| CN111079836A (en) | Process data fault classification method based on pseudo label method and weak supervised learning | |
| CN113536676B (en) | Lithium battery health condition monitoring method based on feature transfer learning | |
| CN102663495B (en) | Neural net data generation method for nonlinear device modeling | |
| CN108491925A (en) | The extensive method of deep learning feature based on latent variable model | |
| CN112132430A (en) | Reliability evaluation method and system for distributed state sensor of power distribution main equipment | |
| Huang et al. | A data-driven method for hybrid data assimilation with multilayer perceptron | |
| CN110262954A (en) | Method based on the automatic learning system reliability model of Condition Monitoring Data | |
| CN111488498A (en) | Node-graph cross-layer graph matching method and system based on graph neural network | |
| CN110442911A (en) | A kind of higher-dimension complication system Uncertainty Analysis Method based on statistical machine learning | |
| CN103279030B (en) | Dynamic soft measuring modeling method and device based on Bayesian frame | |
| CN113033104B (en) | Lithium battery state of charge estimation method based on graph convolution | |
| CN106202926B (en) | Space system deviation based on multi-node collaborative detection is registrated optimization method | |
| CN115031794B (en) | Novel gas-solid two-phase flow measuring method based on multi-feature graph convolution | |
| CN111210877A (en) | Method and device for deducing physical property parameters | |
| Qin et al. | Model identification and Q-matrix incremental inference in cognitive diagnosis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |