CN113052313A - Mass traffic data knowledge mining and parallel processing method - Google Patents

Mass traffic data knowledge mining and parallel processing method Download PDF

Info

Publication number
CN113052313A
CN113052313A CN202110456757.1A CN202110456757A CN113052313A CN 113052313 A CN113052313 A CN 113052313A CN 202110456757 A CN202110456757 A CN 202110456757A CN 113052313 A CN113052313 A CN 113052313A
Authority
CN
China
Prior art keywords
matrix
compression
data
training
flight
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110456757.1A
Other languages
Chinese (zh)
Other versions
CN113052313B (en
Inventor
曹先彬
刘洪岩
朱熙
佟路
杜文博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN202110456757.1A priority Critical patent/CN113052313B/en
Publication of CN113052313A publication Critical patent/CN113052313A/en
Application granted granted Critical
Publication of CN113052313B publication Critical patent/CN113052313B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/29Geographical information databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/044Recurrent networks, e.g. Hopfield networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/04Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10Services
    • G06Q50/26Government or public services

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Economics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Tourism & Hospitality (AREA)
  • Strategic Management (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Human Resources & Organizations (AREA)
  • Evolutionary Computation (AREA)
  • Development Economics (AREA)
  • Marketing (AREA)
  • Databases & Information Systems (AREA)
  • General Business, Economics & Management (AREA)
  • Game Theory and Decision Science (AREA)
  • Remote Sensing (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Educational Administration (AREA)
  • Primary Health Care (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention discloses a method for mining and parallel processing mass traffic data knowledge, which stores an LSTM model in N computing servers of a distributed cluster, divides a training data set into N sets and inputs the N sets into the N computing servers respectively, and trains the computing servers simultaneously, thereby reducing the training time of the LSTM model. After the calculation server finishes one-time forward propagation and backward propagation, the parameter matrix is transmitted to the parameter server and is decomposed into a form of multiplying the two matrixes by adopting a matrix decomposition method, so that the total quantity of the parameters is reduced, and the communication time consumption is reduced. And setting a matrix compression rate, compressing the matrix by using a self-adaptive threshold filtering method, reducing the number of parameters again, and reducing the communication time again. And calculating error matrixes before and after matrix compression and transmitting the error matrixes to the next round of training, calculating the matrixes after errors are eliminated by using the parameter matrixes and the error matrixes when the next round of training is started, and then training the matrixes to compensate errors caused by matrix compression and ensure the accuracy of the LSTM model.

Description

Mass traffic data knowledge mining and parallel processing method
Technical Field
The invention relates to the technical field of traffic big data, in particular to a method for mining and parallel processing mass traffic data knowledge.
Background
Under the background that the informatization level of the traffic system of China is continuously improved, massive traffic big data is continuously generated, and the traffic big data has the characteristics of multi-source heterogeneity, low knowledge density and the like. How to analyze and utilize the traffic big data and mine the knowledge of the traffic big data so as to generate a guiding function for traffic business is a difficult problem which needs to be solved urgently.
Taking the civil aviation industry in China as an example, with the rapid development of the civil aviation industry in China, the airport passenger traffic volume increases year by year, and according to the data statistics of the civil aviation administration in China, the national passenger transport airline company in 2019 executes flights 461.11 ten thousand times, wherein the normal flights 376.52 ten thousand times, and the average normal rate of the flights is 81.65%. The major airline company performed 330.47 ten thousand flights in 2019, with 269.11 ten thousand flights being normal, and the average flight normal rate was 81.43%. The increasing number of flights causes the possibility of flight congestion and collision in the airspace in China to continuously rise, and if an algorithm capable of accurately and quickly predicting the flight path of the flight in a future period is developed, a controller or a pilot can predict the existence of danger in advance before the flight congestion and collision occur, so that the situation can be avoided in time, the danger is greatly avoided, and the flight safety can be ensured, and the life and property safety of passengers can be protected.
Although scholars at home and abroad make great development in the technical field of flight path prediction by knowledge mining of big aviation data at present, the model training needs too long time and cannot reach the rapid standard, and the slow training process leads to the fact that the flight big flight data cannot be utilized to construct an accurate and efficient flight path online incremental prediction model, which means that the availability of the prediction model is low. Therefore, in order to fully utilize knowledge obtained by mining from flight big data and realize accurate and online incremental prediction of flight paths, it is urgently needed to greatly reduce the training time of a prediction model and provide key support for improving the practicability of the flight path prediction model.
Disclosure of Invention
In view of the above, the invention provides a method for mining and parallel processing mass traffic data knowledge, which is used for overcoming the defects that the existing large neural network for predicting flight track points consumes too long time in the training process and cannot effectively perform online incremental prediction.
The invention provides a method for mining and parallel processing mass traffic data knowledge, which comprises the following steps:
s1: acquiring ADS-B data of a flight to be detected in real time, extracting navigation data of six fields in total, namely flight number, position longitude, position latitude, flight altitude, course angle and track point data updating time, from the ADS-B data, and deleting the rest fields;
s2: selecting navigation data of six continuous time slices from the extracted data, inputting the navigation data into an input layer of a flight track prediction model trained in advance, and outputting a prediction result of the navigation data of the flight to be tested through forward propagation;
the training process of the flight path prediction model comprises the following steps:
s11: designing a flight path prediction model, wherein model parameters are designed as follows: the number of nodes of an input layer is set to be 6, the number of nodes of an output layer is set to be 1, the prediction time step length is set to be 6, the number of the hidden layers is set to be 1, the number of nodes of the hidden layers is set to be 60, the training round number is set to be 50, the ReLU is used as an activation function, the cross entropy function is used as a loss function, and the random gradient descent method is used as an optimization method of the loss function;
s12: setting any one server as a parameter server, using the rest N servers as N computing servers, placing all the servers in a local area network to ensure smooth communication among the servers, setting all the servers as a distributed training cluster, copying N copies of a designed flight path prediction model to be respectively deployed in the N computing servers, wherein one flight path prediction model corresponds to one computing server; wherein N is a positive integer;
s13: acquiring historical ADS-B data of a plurality of flights, and extracting flight data of six fields in total, namely flight number, position longitude, position latitude, flight altitude, course angle and track point data updating time from the historical ADS-B data to serve as training data;
s14: averagely dividing the training data into N sets as training data sets of N computing servers, wherein one set corresponds to one computing server, and the data belonging to the same flight are divided into the same subset in each set; sequencing track point data update time fields belonging to the same subset, regarding two track points with time intervals larger than a time threshold value in all track points belonging to the same flight as the ending point and the starting point of two track sequences, dividing all track points belonging to the same flight into a plurality of different track sequences, and constructing the track points in each track sequence into input vectors
Figure 845039DEST_PATH_IMAGE001
Wherein callsign represents flight number, longitude represents position longitude, latitude represents position latitude, height represents flight altitude, angle represents course angle, and time represents track point data updating time;
s15: starting data input in N calculation servers simultaneously, inputting input vectors of six continuous time slices belonging to the same flight path sequence in each subset of each set into six nodes of an input layer of the corresponding calculation server for forward propagation, outputting a prediction result of navigation data of a seventh time slice by inputting the input vector of one time slice into one node, calculating an error between the output prediction result and actual navigation data, and performing backward propagation on the error to obtain N calculation servers subjected to one-time forward propagation and one-time backward propagationParameter matrix of
Figure 213704DEST_PATH_IMAGE002
Wherein, the subscript t represents that the parameter matrix is obtained by the t-th iterative training;
s16: hypothesis parameter matrix
Figure 638869DEST_PATH_IMAGE002
Is one
Figure 674958DEST_PATH_IMAGE003
The parameter matrix is decomposed by matrix decomposition
Figure 769953DEST_PATH_IMAGE002
Decomposed into three matrices
Figure 891493DEST_PATH_IMAGE004
Figure 369616DEST_PATH_IMAGE005
And
Figure 994633DEST_PATH_IMAGE006
wherein, the matrix
Figure 526108DEST_PATH_IMAGE004
Is one
Figure 869365DEST_PATH_IMAGE007
Of a matrix, a matrix
Figure 901912DEST_PATH_IMAGE008
Is one
Figure 912593DEST_PATH_IMAGE009
Of a matrix, a matrix
Figure 349391DEST_PATH_IMAGE006
Is one
Figure 179943DEST_PATH_IMAGE010
Diagonal moment ofMatrix, matrix
Figure 501334DEST_PATH_IMAGE006
Having non-negative real singular values in descending order, R representing a parameter matrix
Figure 366522DEST_PATH_IMAGE002
The rank of (d); order to
Figure 239800DEST_PATH_IMAGE011
Figure 557649DEST_PATH_IMAGE012
Figure 541786DEST_PATH_IMAGE013
Matrix of
Figure 386114DEST_PATH_IMAGE014
Is one
Figure 430293DEST_PATH_IMAGE015
Of a matrix, a matrix
Figure 235438DEST_PATH_IMAGE016
One is
Figure 23266DEST_PATH_IMAGE017
A matrix of (a);
s17: determining a training pair matrix for each iteration
Figure 974297DEST_PATH_IMAGE018
And
Figure 189378DEST_PATH_IMAGE016
according to the compression ratio, determining each iteration training pair matrix in a self-adaptive way
Figure 216240DEST_PATH_IMAGE018
And
Figure 807758DEST_PATH_IMAGE016
compression threshold of, rootAccording to the compression rate and the compression threshold, respectively corresponding to the matrix
Figure 361099DEST_PATH_IMAGE018
And
Figure 747081DEST_PATH_IMAGE016
the parameter number of the first compression matrix is compressed to obtain a first compression matrix and a second compression matrix, and the first compression matrix and the matrix are compressed
Figure 526819DEST_PATH_IMAGE018
Subtracting the corresponding parameters to obtain a first error matrix, and compressing the second compression matrix with the first error matrix
Figure 656449DEST_PATH_IMAGE016
Subtracting the corresponding parameters to obtain a second error matrix, and inputting the first compression matrix, the second compression matrix, the first error matrix and the second error matrix into the parameter server;
s18: averaging first compression matrixes of N calculation servers to obtain a first average compression matrix, averaging second compression matrixes of N calculation servers to obtain a second average compression matrix, averaging first error matrixes of N calculation servers to obtain a first average error matrix, averaging second error matrixes of N calculation servers to obtain a second average error matrix, and transmitting the first average compression matrix, the second average compression matrix, the first average error matrix and the second average error matrix back to the N calculation servers;
s19: in N calculation servers, a first average compression matrix and a first average error matrix are used for compensating errors generated by the first compression matrix to obtain a first compensation matrix, a second average compression matrix and a second average error matrix are used for compensating errors generated by the second compression matrix to obtain a second compensation matrix, the first compensation matrix and the second compensation matrix are multiplied to obtain an optimized parameter matrix which is used as a new parameter matrix for next iterative training, the step S15 is returned, forward propagation is carried out again by using the new parameter matrix, a new prediction result is output, errors between the new prediction result and actual navigation data are calculated, errors between the new prediction result and the actual navigation data are evaluated by using a loss function, and the training is stopped when the loss function reaches the minimum value or an overfitting phenomenon occurs in the training process.
In a possible implementation manner, in the method for mining and processing the mass traffic data knowledge provided by the present invention, in step S17, the matrices are respectively processed according to the compression rate and the compression threshold
Figure 205242DEST_PATH_IMAGE018
And
Figure 903070DEST_PATH_IMAGE016
the parameter number of the first compression matrix and the parameter number of the second compression matrix are compressed to obtain a first compression matrix and a second compression matrix, and the method specifically comprises the following steps:
will matrix
Figure 904524DEST_PATH_IMAGE019
Setting elements with the values smaller than the compression threshold value corresponding to the row in the vectors of the middle rows as 0, keeping other elements unchanged, obtaining a filter operator Mask matrix, and calculating a first compression matrix and a second compression matrix according to the following formulas:
Figure 837845DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 834620DEST_PATH_IMAGE021
representing a compression matrix, p =1, 2,
Figure 93563DEST_PATH_IMAGE022
representing the multiplication of corresponding elements of two isotypic matrices.
In a possible implementation manner, in the method for mining and parallel processing of the mass traffic data knowledge provided by the invention, in step S17, the compression threshold is updated after each L times of iterative training, and the value range of L is 1000-1500.
The method for mining and processing the mass traffic data knowledge provided by the invention stores the designed LSTM model in each computing server of the distributed cluster, divides the mass training data set into sets with the same number as the computing servers when the training process starts, inputs different sets into different computing servers, and trains the computing servers simultaneously, thereby greatly reducing the overall training time of the LSTM model. After the calculation server finishes one-time forward propagation and backward propagation, the parameter matrix obtained by transmission to the parameter server is decomposed into a form of multiplying the two matrixes by a matrix decomposition method, and the total quantity of the parameters of the two matrixes is far less than the quantity of the parameters of the parameter matrix, so that the quantity of the parameters needing to be transmitted by each calculation server is greatly reduced, and the time consumed by communication is correspondingly reduced. On the basis of the two matrixes obtained by decomposition, the compression rate of the matrixes is set, a filtering threshold value is determined in each row vector of the matrixes by using a self-adaptive threshold value filtering method, corresponding filtering operator Mask matrixes are obtained, the filtering operator Mask matrixes are multiplied by the matrixes obtained by decomposition, the compression matrixes can be obtained, the number of integral parameters is greatly reduced again compared with the number of the integral parameters before, and the communication time consumption is greatly reduced again. After compression of the parameter matrix is carried out, some calculation errors are inevitably introduced, and if the calculation errors are not eliminated, the errors can continuously accumulate and influence the final model precision after multiple rounds of training. The invention calculates the error vector before and after each row vector compression in the matrix, and transmits the error vector formed by the error vector to the next round of training of each calculation server, and when the next round of training begins, firstly, the parameter matrix obtained from the parameter server and the error matrix are used for calculating to obtain the parameter matrix with the error eliminated, and then the subsequent training work is carried out, thereby well compensating the error caused by the matrix compression and keeping the accuracy of the finally obtained model basically unchanged.
Drawings
Fig. 1 is a flowchart of a method for mining and processing mass traffic data knowledge in parallel according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only illustrative and are not intended to limit the present invention.
The invention provides a method for mining and parallel processing mass traffic data knowledge, which comprises the following steps:
s1: acquiring ADS-B data of a flight to be detected in real time, extracting navigation data of six fields in total of flight number, position longitude, position latitude, flight altitude, course angle and track point data updating time from the ADS-B data, and deleting the rest fields;
s2: selecting navigation data of six continuous time slices from the extracted data, inputting the navigation data into an input layer of a flight track prediction model trained in advance, and outputting a prediction result of the navigation data of the flight to be tested through forward propagation;
the training process of the flight path prediction model comprises the following steps:
s11: designing a flight path prediction model, wherein model parameters are designed as follows: the number of nodes of an input layer is set to be 6, the number of nodes of an output layer is set to be 1, the prediction time step length is set to be 6, the number of the hidden layers is set to be 1, the number of nodes of the hidden layers is set to be 60, the training round number is set to be 50, the ReLU is used as an activation function, the cross entropy function is used as a loss function, and the random gradient descent method is used as an optimization method of the loss function;
s12: setting any one server as a parameter server, using the rest N servers as N computing servers, placing all the servers in a local area network to ensure smooth communication among the servers, setting all the servers as a distributed training cluster, copying N copies of a designed flight path prediction model to be respectively deployed in the N computing servers, wherein one flight path prediction model corresponds to one computing server; wherein N is a positive integer;
s13: acquiring historical ADS-B data of a plurality of flights, and extracting flight data of six fields in total, such as flight numbers, position longitudes, position latitudes, flight altitudes, course angles and track point data updating time, from the historical ADS-B data to serve as training data;
s14: the training data is divided equally into N sets,as training data sets of N computing servers, one set corresponds to one computing server, and data belonging to the same flight are divided into the same subset in each set; sequencing track point data update time fields belonging to the same subset, regarding two track points with time intervals larger than a time threshold value in all track points belonging to the same flight as the ending point and the starting point of two track sequences, dividing all track points belonging to the same flight into a plurality of different track sequences, and constructing the track points in each track sequence into input vectors
Figure 582313DEST_PATH_IMAGE023
Wherein callsign represents flight number, longitude represents position longitude, latitude represents position latitude, height represents flight altitude, angle represents course angle, and time represents track point data updating time;
s15: starting data input in N calculation servers simultaneously, inputting input vectors of six continuous time slices belonging to the same flight path sequence in each subset of each set into six nodes of an input layer of the corresponding calculation server for forward propagation, outputting a prediction result of navigation data of a seventh time slice, calculating an error between the output prediction result and actual navigation data, and performing backward propagation on the error to obtain a parameter matrix of the N calculation servers after one-time forward propagation and one-time backward propagation
Figure 53746DEST_PATH_IMAGE002
Wherein, the subscript t represents that the parameter matrix is obtained by the t-th iterative training;
s16: hypothesis parameter matrix
Figure 311552DEST_PATH_IMAGE002
Is one
Figure 849718DEST_PATH_IMAGE003
The parameter matrix is decomposed by matrix decomposition
Figure 560185DEST_PATH_IMAGE002
Decomposed into three matrices
Figure 835309DEST_PATH_IMAGE004
Figure 213201DEST_PATH_IMAGE005
And
Figure 407422DEST_PATH_IMAGE006
wherein, the matrix
Figure 870764DEST_PATH_IMAGE004
Is one
Figure 683999DEST_PATH_IMAGE007
Of a matrix, a matrix
Figure 916398DEST_PATH_IMAGE008
Is one
Figure 32252DEST_PATH_IMAGE009
Of a matrix, a matrix
Figure 982891DEST_PATH_IMAGE006
Is one
Figure 599817DEST_PATH_IMAGE010
Of a diagonal matrix, matrix
Figure 686722DEST_PATH_IMAGE006
Having non-negative real singular values in descending order, R representing a parameter matrix
Figure 488324DEST_PATH_IMAGE002
The rank of (d); order to
Figure 926259DEST_PATH_IMAGE011
Figure 81297DEST_PATH_IMAGE012
Figure 288287DEST_PATH_IMAGE013
Matrix of
Figure 870578DEST_PATH_IMAGE014
Is one
Figure 629763DEST_PATH_IMAGE015
Of a matrix, a matrix
Figure 588491DEST_PATH_IMAGE016
One is
Figure 649988DEST_PATH_IMAGE017
A matrix of (a);
s17: determining a training pair matrix for each iteration
Figure 934339DEST_PATH_IMAGE018
And
Figure 205921DEST_PATH_IMAGE016
according to the compression ratio, determining each iteration training pair matrix in a self-adaptive way
Figure 702761DEST_PATH_IMAGE018
And
Figure 618764DEST_PATH_IMAGE016
according to the compression ratio and the compression threshold, respectively for the matrix
Figure 808437DEST_PATH_IMAGE018
And
Figure 318047DEST_PATH_IMAGE016
the parameter number of the first compression matrix is compressed to obtain a first compression matrix and a second compression matrix, and the first compression matrix and the matrix are compressed
Figure 618578DEST_PATH_IMAGE018
Subtracting the corresponding parameters to obtain a first error matrix, and compressing the second compression matrix with the first error matrix
Figure 389088DEST_PATH_IMAGE016
Subtracting the corresponding parameters to obtain a second error matrix, and inputting the first compression matrix, the second compression matrix, the first error matrix and the second error matrix into the parameter server;
s18: averaging first compression matrixes of N calculation servers to obtain a first average compression matrix, averaging second compression matrixes of N calculation servers to obtain a second average compression matrix, averaging first error matrixes of N calculation servers to obtain a first average error matrix, averaging second error matrixes of N calculation servers to obtain a second average error matrix, and transmitting the first average compression matrix, the second average compression matrix, the first average error matrix and the second average error matrix back to the N calculation servers;
s19: in N calculation servers, a first average compression matrix and a first average error matrix are used for compensating errors generated by the first compression matrix to obtain a first compensation matrix, a second average compression matrix and a second average error matrix are used for compensating errors generated by the second compression matrix to obtain a second compensation matrix, the first compensation matrix and the second compensation matrix are multiplied to obtain an optimized parameter matrix which is used as a new parameter matrix for next iterative training, the step S15 is returned, forward propagation is carried out again by using the new parameter matrix, a new prediction result is output, errors between the new prediction result and actual navigation data are calculated, errors between the new prediction result and the actual navigation data are evaluated by using a loss function, and the training is stopped when the loss function reaches the minimum value or an overfitting phenomenon occurs in the training process.
In specific implementation, when step S17 of the method for mining and processing mass traffic data knowledge provided by the present invention is executed, the matrix is respectively processed according to the compression rate and the compression threshold
Figure 749663DEST_PATH_IMAGE018
And
Figure 995836DEST_PATH_IMAGE016
parameter (d) ofThe number is compressed to obtain a first compression matrix and a second compression matrix, which can be specifically realized by the following method: will matrix
Figure 834479DEST_PATH_IMAGE024
Setting elements with the values smaller than the compression threshold value corresponding to the row in the vectors of the middle rows as 0, keeping other elements unchanged, obtaining a filter operator Mask matrix, and calculating a first compression matrix and a second compression matrix according to the following formulas:
Figure 725075DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 256550DEST_PATH_IMAGE021
representing a compression matrix, p =1, 2,
Figure 973708DEST_PATH_IMAGE025
representing the multiplication of corresponding elements of two isotypic matrices.
In specific implementation, when step S17 in the method for mining and parallel processing of mass traffic data knowledge provided by the present invention is executed, the compression threshold is updated after L iterative training, and preferably, the value range of L may be 1000 to 1500.
The following describes a specific implementation of the knowledge mining and parallel processing method for mass traffic data according to the present invention in detail by using a specific embodiment.
Example 1: the method comprises a model training process and an actual prediction process.
Model training procedure
The first step is as follows: construction of flight path prediction model
The flight path prediction model is designed by LSTM, and the specific parameters of the model are designed as follows: the number of nodes of an input layer is set to be 6, the number of nodes of an output layer is set to be 1, the prediction time step length is set to be 6, the number of the hidden layers is set to be 1, the number of nodes of the hidden layers is set to be 60, the training round is set to be 50, the ReLU is used as an activation function, the cross entropy function is used as a loss function, and the random gradient descent method is used as the optimization method of the loss function.
The second step is that: building input data
Acquiring historical ADS-B data of a plurality of flights, and extracting flight data of six fields in total, namely flight number, position longitude, position latitude, flight altitude, course angle and track point data updating time, from the historical ADS-B data to serve as training data. Dividing all training data into N (N is a positive integer) sets on average, using the N sets as training data sets of N computing servers, wherein one set corresponds to one computing server, and data belonging to the same flight are subdivided into the same subset in each set and recorded as
Figure 881621DEST_PATH_IMAGE026
. Then, for the same subset
Figure 626723DEST_PATH_IMAGE027
The track point data updating time fields in the flight path are sequenced, and two track points with time intervals larger than a time threshold value in all track points belonging to the same flight are taken as the ending point and the starting point of two track sequences, so that all track points belonging to the same flight can be divided into a plurality of different track sequences and recorded as different track sequences
Figure 594679DEST_PATH_IMAGE028
. Constructing track points in each track sequence into input vectors
Figure 425232DEST_PATH_IMAGE023
Wherein callsign represents flight number, longitude represents position longitude, latitude represents position latitude, height represents flight altitude, angle represents heading angle, and time represents data update time.
The training data set is divided into sets with the same number as the computing servers, different sets are input into different computing servers, and all the servers carry out training simultaneously, so that the overall training time of the LSTM model can be greatly reduced.
The third step: inputting training data into flight path prediction model
Any one server is set as a parameter server, the rest N servers are set as N computing servers, and all the servers are placed in a local area network to ensure smooth communication among the servers. All the servers are set as distributed training clusters, N copies of a designed flight path prediction model (LSTM) are copied and deployed to N computing servers respectively, and one flight path prediction model corresponds to one computing server.
Simultaneously starting data input in N computing servers, and enabling each subset of each set to belong to the same track sequence
Figure 730312DEST_PATH_IMAGE029
The input vectors of six continuous time slices are input into six nodes of an input layer corresponding to the calculation servers for forward propagation, the input vector of one time slice corresponds to one node, a prediction result of navigation data of a seventh time slice is output, the error between the output prediction result and actual navigation data is calculated, the error is propagated in the reverse direction, and parameter matrixes of N calculation servers after one-time forward propagation and one-time backward propagation are obtained
Figure 595499DEST_PATH_IMAGE002
And the subscript t indicates that the parameter matrix is obtained by the t-th iteration training.
The fourth step: parameter matrix
Figure 468778DEST_PATH_IMAGE002
Decomposed into two smaller matrices
Figure 786626DEST_PATH_IMAGE018
And
Figure 646129DEST_PATH_IMAGE016
hypothesis parameter matrix
Figure 365823DEST_PATH_IMAGE002
Is one
Figure 410003DEST_PATH_IMAGE003
The matrix adopts a matrix decomposition method to matrix the parameters with more parameters
Figure 215148DEST_PATH_IMAGE002
Decomposed into three matrices
Figure 127609DEST_PATH_IMAGE004
Figure 701810DEST_PATH_IMAGE008
And
Figure 916891DEST_PATH_IMAGE006
wherein, the matrix
Figure 943752DEST_PATH_IMAGE004
Is one
Figure 912102DEST_PATH_IMAGE030
Of a matrix, a matrix
Figure 340809DEST_PATH_IMAGE008
Is one
Figure 726791DEST_PATH_IMAGE009
Of a matrix, a matrix
Figure 506528DEST_PATH_IMAGE006
Is one
Figure 760792DEST_PATH_IMAGE010
Of a diagonal matrix, matrix
Figure 44006DEST_PATH_IMAGE006
Having non-negative real singular values in descending order, R representing a parameter matrix
Figure 132048DEST_PATH_IMAGE002
Is determined. Entire parameter matrix
Figure 133502DEST_PATH_IMAGE002
The formula for decomposition can be written as:
Figure 801243DEST_PATH_IMAGE031
(1)
expressed by the formula (1), a parameter matrix
Figure 79909DEST_PATH_IMAGE002
Can be written as two smaller matrices
Figure 73273DEST_PATH_IMAGE018
And
Figure 296444DEST_PATH_IMAGE016
form of multiplication, i.e.
Figure 33456DEST_PATH_IMAGE032
Wherein, in the step (A),
Figure 415895DEST_PATH_IMAGE018
is equal to
Figure 314581DEST_PATH_IMAGE033
Figure 290628DEST_PATH_IMAGE016
Is equal to
Figure 565751DEST_PATH_IMAGE034
. Original parameter matrix
Figure 51965DEST_PATH_IMAGE002
The number of parameters in (1) is
Figure 387131DEST_PATH_IMAGE003
One, and the two matrices after decomposition
Figure 584894DEST_PATH_IMAGE018
And
Figure 663709DEST_PATH_IMAGE016
the number of parameters is reduced by a large amount, matrix
Figure 20741DEST_PATH_IMAGE018
The number of parameters in (1) is
Figure 261229DEST_PATH_IMAGE015
A matrix of
Figure 211868DEST_PATH_IMAGE016
The number of parameters in (1) is
Figure 828794DEST_PATH_IMAGE009
If one, the sum of the parameter numbers in the two matrixes is
Figure 791065DEST_PATH_IMAGE035
And (4) respectively. If R is much smaller than the minimum of m and n, it is certain that the sum of the number of parameters of the decomposed two matrices is much smaller than the number of parameters in the original parameter matrix, i.e. R is much smaller than the minimum of m and n
Figure 202455DEST_PATH_IMAGE036
Therefore, the number of parameters to be transmitted by each computing server is greatly reduced, and the time consumed by communication is correspondingly reduced. In the actual training process, the original parameter matrix is found through testing
Figure 374810DEST_PATH_IMAGE002
In (1),
Figure 795427DEST_PATH_IMAGE037
Figure 2418DEST_PATH_IMAGE038
if it is present
Figure 974922DEST_PATH_IMAGE002
Is 64, i.e.
Figure 634573DEST_PATH_IMAGE039
The matrix decomposition method can convert the original parameter matrix
Figure 593302DEST_PATH_IMAGE002
The 5000000 parameters are reduced to 300000, and the number of parameters needing to be transmitted is greatly reduced.
The fifth step: decomposing the matrix
Figure 654799DEST_PATH_IMAGE018
And
Figure 50401DEST_PATH_IMAGE016
further compression
Matrix after decomposition
Figure 931770DEST_PATH_IMAGE018
And
Figure 694189DEST_PATH_IMAGE016
compressing the parameter number before transmitting to the network constructed by the calculation server and the parameter server, firstly determining the training pair matrix of each iteration
Figure 610193DEST_PATH_IMAGE018
And
Figure 924499DEST_PATH_IMAGE016
according to the compression ratio, determining each iteration training pair matrix in a self-adaptive way
Figure 558743DEST_PATH_IMAGE018
And
Figure 859274DEST_PATH_IMAGE016
according to the compression ratio and the compression threshold, respectively for the matrix
Figure 629784DEST_PATH_IMAGE018
And
Figure 255938DEST_PATH_IMAGE016
parameter (d) ofThe number of the first compression matrix and the second compression matrix can be obtained by compressing, compared with the number of the whole parameters before compression, the number of the whole parameters is greatly reduced again, and the communication time consumption is greatly reduced again. By reducing the total number of parameters communicated in a network constructed by the calculation server and the parameter server, the communication overhead among different servers in the training process is reduced as much as possible, so that the total training time is reduced.
For the two matrices obtained by the decomposition in the fourth step
Figure 987264DEST_PATH_IMAGE018
And
Figure 91487DEST_PATH_IMAGE016
the compression process is the same, and thus, here, the matrix is referred to as a pair matrix
Figure 982082DEST_PATH_IMAGE018
The compression is used as an example to illustrate the matrix
Figure 513558DEST_PATH_IMAGE016
The compression process is similar and will not be described herein.
First, the compression ratio is determined
Figure 981448DEST_PATH_IMAGE040
Since after the compression ratio is determined, the matrix is trained every iteration
Figure 623782DEST_PATH_IMAGE014
The Kth value of each row vector does not change much, wherein
Figure 368884DEST_PATH_IMAGE041
Therefore, it is not reasonable to update the compression threshold every iteration of training. In compressing the matrix
Figure 71261DEST_PATH_IMAGE018
The compression threshold is updated after each L iterative training, where L may be 1000 times. When it is firstIn the secondary iteration training, a compression threshold value needs to be calculated, and the matrix is used
Figure 275715DEST_PATH_IMAGE018
Each row of (a) is regarded as m R-dimensional vectors, values in each vector are arranged in a descending order, and a formula is expressed as:
Figure 721740DEST_PATH_IMAGE042
(2)
wherein the content of the first and second substances,
Figure 586928DEST_PATH_IMAGE043
representing a matrix after descending order
Figure 460206DEST_PATH_IMAGE018
The 1 st row vector of the first row vector,
Figure 637109DEST_PATH_IMAGE044
representing a matrix after descending order
Figure 886825DEST_PATH_IMAGE014
The 2 nd row vector of the second row vector,
Figure 606519DEST_PATH_IMAGE045
representing a matrix after descending order
Figure 650699DEST_PATH_IMAGE018
The m-th row vector of the vector,
Figure 65631DEST_PATH_IMAGE046
representation pair matrix
Figure 853458DEST_PATH_IMAGE014
The elements in each row vector are arranged in descending order, each element in the vector on the left side of the equation is a main target of the subsequent operation, and each element is an R-dimensional vector. Preserving the inner prefixes of each vector
Figure 427659DEST_PATH_IMAGE047
Individual element, can determine the securityThe smallest of the elements left is the compression threshold of the vector, from which the entire matrix can be derived
Figure 642740DEST_PATH_IMAGE014
The compression threshold vector of (a) is as follows:
Figure 59815DEST_PATH_IMAGE048
(3)
wherein the content of the first and second substances,
Figure 651333DEST_PATH_IMAGE049
representation matrix
Figure 345619DEST_PATH_IMAGE018
The compression threshold of the 1 st row vector,
Figure 731601DEST_PATH_IMAGE050
representation matrix
Figure 245759DEST_PATH_IMAGE018
The compression threshold of the 2 nd row vector,
Figure 486641DEST_PATH_IMAGE051
representation matrix
Figure 35434DEST_PATH_IMAGE018
A compression threshold for the m-th row vector; in the subsequent iterative training process, if the iteration times are not integral multiples of L, the compression threshold vector used in the previous iterative training can be used; if the iteration times are integral multiples of L, the compressed threshold vector needs to be updated, and the calculation method is consistent with the first calculation of the compressed threshold vector. Calculating a Mask matrix of filter operators, and calculating the matrix
Figure 857897DEST_PATH_IMAGE018
And setting the element with the numerical value smaller than the compression threshold value corresponding to the row in each row vector as 0, and keeping other elements unchanged to obtain a matrix which is the Mask matrix of the filter operator. Will matrix
Figure 859351DEST_PATH_IMAGE018
Multiplying the filter operator Mask matrix corresponding elements one by one to obtain a compressed matrix, namely a first compressed matrix
Figure 917305DEST_PATH_IMAGE052
The calculation formula can be written as:
Figure 320605DEST_PATH_IMAGE053
(4)
wherein the content of the first and second substances,
Figure 48390DEST_PATH_IMAGE022
representing the multiplication of corresponding elements of two isotypic matrices.
And a sixth step: compensating compression error and performing communication between servers
After compression of the parameter matrix is carried out, some calculation errors are inevitably introduced, and if the calculation errors are not eliminated, the errors can continuously accumulate and influence the final model precision after multiple rounds of training.
The first compression matrix obtained in the fifth step is processed
Figure 537140DEST_PATH_IMAGE052
Sum matrix
Figure 149518DEST_PATH_IMAGE018
By subtraction, the quantization error after and before compression, i.e. the AND matrix, can be obtained
Figure 407324DEST_PATH_IMAGE018
And a first compression matrix
Figure 306010DEST_PATH_IMAGE052
First error matrix of the same type
Figure 282056DEST_PATH_IMAGE054
Here, subtraction means subtraction of corresponding elements between two matrices. At this time obtainTo two matrices participating in the communication
Figure 681813DEST_PATH_IMAGE054
And
Figure 794126DEST_PATH_IMAGE052
matrix of
Figure 129292DEST_PATH_IMAGE054
And
Figure 327055DEST_PATH_IMAGE052
is compared with the matrix
Figure 779771DEST_PATH_IMAGE018
The number of parameters is greatly reduced, and the communication time can be greatly reduced. All the calculation servers transmit the first compression matrix and the first error matrix to the parameter server after obtaining the respective first compression matrix and the first error matrix, if some calculation servers do not finish the calculation after transmitting, other calculation servers synchronously wait, and when the parameter server receives the first compression matrices of all the N calculation servers
Figure 746590DEST_PATH_IMAGE052
And a first error matrix
Figure 518237DEST_PATH_IMAGE055
Then, the first compression matrix of the N computing servers is used
Figure 203296DEST_PATH_IMAGE052
And a first error matrix
Figure 820222DEST_PATH_IMAGE054
Respectively calculating the average values to obtain a first average error matrix
Figure 31761DEST_PATH_IMAGE056
And a first average compression matrix
Figure 708730DEST_PATH_IMAGE057
The calculation formula is as follows:
Figure 881085DEST_PATH_IMAGE058
(5)
Figure 911489DEST_PATH_IMAGE059
(6)
in the above two formulas, the calculation between the matrices is the operation between the corresponding elements of the same type of matrix. Obtaining a first average compression matrix
Figure 852900DEST_PATH_IMAGE056
And a first average error matrix
Figure 700771DEST_PATH_IMAGE057
Then, the first average compression matrix is
Figure 360422DEST_PATH_IMAGE056
And a first average error matrix
Figure 443785DEST_PATH_IMAGE057
And simultaneously transmitted back to the N computing servers. In N calculation servers, a first average compression matrix and a first average error matrix are used for compensating errors generated by the first compression matrix to obtain a first compensation matrix, the first compensation matrix and a second compensation matrix are multiplied to obtain an optimized (namely error-eliminated) parameter matrix, and the optimized parameter matrix is used as a new parameter matrix for next iterative training, so that errors caused by matrix compression can be well compensated, and finally obtained model precision is basically kept unchanged.
The seventh step: end of training
And returning to the third step, carrying out forward propagation again by using a new parameter matrix, outputting a new prediction result, calculating the error between the new prediction result and the actual navigation data, evaluating the error between the new prediction result and the actual navigation data by using a loss function, and stopping training when the loss function reaches the minimum value or an overfitting phenomenon occurs in the training process. A trained flight path prediction model (LSTM) can be used to actually predict flight path point positions. After the flight path prediction model is trained,
(II) actual prediction Process
The first step is as follows: flight big data preprocessing
Acquiring flight ADS-B data of domestic flights, wherein the data comprises flight numbers, take-off airports, take-off airport longitudes and latitudes, landing airports, landing airport longitudes and latitudes, planned take-off time, planned arrival time, planned routes, position longitudes and latitudes, flight heights, course angles and data updating time. And (3) carrying out data cleaning on the flight schedule data, extracting six fields of flight number, position longitude, position latitude, flight altitude, course angle and data updating time in the original data table, and deleting the rest fields.
The second step is that: actual prediction
Selecting navigation data of six continuous time slices from the extracted data, inputting the navigation data into an input layer of a flight track prediction model trained in advance, and outputting a prediction result of the navigation data of the flight to be tested through forward propagation.
The method for mining and processing the mass traffic data knowledge provided by the invention stores the designed LSTM model in each computing server of the distributed cluster, divides the mass training data set into sets with the same number as the computing servers when the training process starts, inputs different sets into different computing servers, and trains the computing servers simultaneously, thereby greatly reducing the overall training time of the LSTM model. After the calculation server finishes one-time forward propagation and backward propagation, the parameter matrix obtained by transmission to the parameter server is decomposed into a form of multiplying two matrixes by a matrix decomposition method, and the total quantity of the parameters of the two matrixes is far less than the quantity of the parameters of the parameter matrix, so that the quantity of the parameters needing to be transmitted by each calculation server is greatly reduced, and the time consumed by communication is correspondingly reduced. On the basis of the two matrixes obtained by decomposition, the compression rate of the matrixes is set, a filtering threshold value is determined in each row vector of the matrixes by using a self-adaptive threshold value filtering method, corresponding filtering operator Mask matrixes are obtained, the filtering operator Mask matrixes are multiplied by the matrixes obtained by decomposition, the compression matrixes can be obtained, the number of integral parameters is greatly reduced again compared with the number of the integral parameters, and the communication time consumption is greatly reduced again. After compression of the parameter matrix is carried out, some calculation errors are inevitably introduced, and if the calculation errors are not eliminated, the errors can continuously accumulate and influence the final model precision after multiple rounds of training. The invention calculates the error vector before and after each row vector compression in the matrix, and transmits the error vector formed by the error vector to the next round of training of each calculation server, and when the next round of training begins, firstly, the parameter matrix obtained from the parameter server and the error matrix are used for calculating to obtain the parameter matrix with the error eliminated, and then the subsequent training work is carried out, thereby well compensating the error caused by the matrix compression and keeping the accuracy of the finally obtained model basically unchanged.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims (3)

1. A method for mining and parallel processing mass traffic data knowledge is characterized by comprising the following steps:
s1: acquiring ADS-B data of a flight to be detected in real time, extracting navigation data of six fields in total, namely flight number, position longitude, position latitude, flight altitude, course angle and track point data updating time, from the ADS-B data, and deleting the rest fields;
s2: selecting navigation data of six continuous time slices from the extracted data, inputting the navigation data into an input layer of a flight track prediction model trained in advance, and outputting a prediction result of the navigation data of the flight to be tested through forward propagation;
the training process of the flight path prediction model comprises the following steps:
s11: designing a flight path prediction model, wherein model parameters are designed as follows: the number of nodes of an input layer is set to be 6, the number of nodes of an output layer is set to be 1, the prediction time step length is set to be 6, the number of the hidden layers is set to be 1, the number of nodes of the hidden layers is set to be 60, the training round number is set to be 50, the ReLU is used as an activation function, the cross entropy function is used as a loss function, and the random gradient descent method is used as an optimization method of the loss function;
s12: setting any one server as a parameter server, using the rest N servers as N computing servers, placing all the servers in a local area network to ensure smooth communication among the servers, setting all the servers as a distributed training cluster, copying N copies of a designed flight path prediction model to be respectively deployed in the N computing servers, wherein one flight path prediction model corresponds to one computing server; wherein N is a positive integer;
s13: acquiring historical ADS-B data of a plurality of flights, and extracting flight data of six fields in total, namely flight number, position longitude, position latitude, flight altitude, course angle and track point data updating time from the historical ADS-B data to serve as training data;
s14: averagely dividing the training data into N sets as training data sets of N computing servers, wherein one set corresponds to one computing server, and the data belonging to the same flight are divided into the same subset in each set; sequencing track point data update time fields belonging to the same subset, regarding two track points with time intervals larger than a time threshold value in all track points belonging to the same flight as the ending point and the starting point of two track sequences, dividing all track points belonging to the same flight into a plurality of different track sequences, and constructing the track points in each track sequence into input vectors
Figure 222302DEST_PATH_IMAGE001
Wherein callsign represents flight number, longitude represents position longitude, latitude represents position latitude, height represents flight altitude, angle represents course angle, and time represents track point data updating time;
s15: simultaneously starting data entry in N computing serversInputting the input vectors of six continuous time slices belonging to the same track sequence in each subset of each set into six nodes of an input layer of a corresponding calculation server for forward propagation, outputting a prediction result of navigation data of a seventh time slice by inputting the input vector of one time slice into one node, calculating the error between the output prediction result and actual navigation data, and performing backward propagation on the error to obtain a parameter matrix of N calculation servers subjected to one-time forward propagation and one-time backward propagation
Figure 873863DEST_PATH_IMAGE002
Wherein, the subscript t represents that the parameter matrix is obtained by the t-th iterative training;
s16: hypothesis parameter matrix
Figure 388021DEST_PATH_IMAGE002
Is one
Figure 642285DEST_PATH_IMAGE003
The parameter matrix is decomposed by matrix decomposition
Figure 191078DEST_PATH_IMAGE002
Decomposed into three matrices
Figure 747961DEST_PATH_IMAGE004
Figure 749415DEST_PATH_IMAGE005
And
Figure 71286DEST_PATH_IMAGE006
wherein, the matrix
Figure 474585DEST_PATH_IMAGE004
Is one
Figure 202370DEST_PATH_IMAGE007
Of a matrix, a matrix
Figure 691120DEST_PATH_IMAGE008
Is one
Figure 287186DEST_PATH_IMAGE009
Of a matrix, a matrix
Figure 544992DEST_PATH_IMAGE006
Is one
Figure 709257DEST_PATH_IMAGE010
Of a diagonal matrix, matrix
Figure 685304DEST_PATH_IMAGE006
Having non-negative real singular values in descending order, R representing a parameter matrix
Figure 570214DEST_PATH_IMAGE002
The rank of (d); order to
Figure 948106DEST_PATH_IMAGE011
Figure 283272DEST_PATH_IMAGE012
Figure 746615DEST_PATH_IMAGE013
Matrix of
Figure 418905DEST_PATH_IMAGE014
Is one
Figure 651303DEST_PATH_IMAGE015
Of a matrix, a matrix
Figure 422950DEST_PATH_IMAGE016
One is
Figure 108009DEST_PATH_IMAGE017
A matrix of (a);
s17: determining a training pair matrix for each iteration
Figure 724935DEST_PATH_IMAGE018
And
Figure 185741DEST_PATH_IMAGE016
according to the compression ratio, determining each iteration training pair matrix in a self-adaptive way
Figure 862710DEST_PATH_IMAGE018
And
Figure 35065DEST_PATH_IMAGE016
according to the compression ratio and the compression threshold, respectively for the matrix
Figure 190103DEST_PATH_IMAGE018
And
Figure 521728DEST_PATH_IMAGE016
the parameter number of the first compression matrix is compressed to obtain a first compression matrix and a second compression matrix, and the first compression matrix and the matrix are compressed
Figure 369598DEST_PATH_IMAGE018
Subtracting the corresponding parameters to obtain a first error matrix, and compressing the second compression matrix with the first error matrix
Figure 29249DEST_PATH_IMAGE016
Subtracting the corresponding parameters to obtain a second error matrix, and inputting the first compression matrix, the second compression matrix, the first error matrix and the second error matrix into the parameter server;
s18: averaging first compression matrixes of N calculation servers to obtain a first average compression matrix, averaging second compression matrixes of N calculation servers to obtain a second average compression matrix, averaging first error matrixes of N calculation servers to obtain a first average error matrix, averaging second error matrixes of N calculation servers to obtain a second average error matrix, and transmitting the first average compression matrix, the second average compression matrix, the first average error matrix and the second average error matrix back to the N calculation servers;
s19: in N calculation servers, a first average compression matrix and a first average error matrix are used for compensating errors generated by the first compression matrix to obtain a first compensation matrix, a second average compression matrix and a second average error matrix are used for compensating errors generated by the second compression matrix to obtain a second compensation matrix, the first compensation matrix and the second compensation matrix are multiplied to obtain an optimized parameter matrix which is used as a new parameter matrix for next iterative training, the step S15 is returned, forward propagation is carried out again by using the new parameter matrix, a new prediction result is output, errors between the new prediction result and actual navigation data are calculated, errors between the new prediction result and the actual navigation data are evaluated by using a loss function, and the training is stopped when the loss function reaches the minimum value or an overfitting phenomenon occurs in the training process.
2. The method for knowledge mining and parallel processing of mass traffic data according to claim 1, wherein in step S17, the matrices are respectively processed according to compression rate and compression threshold
Figure 597765DEST_PATH_IMAGE018
And
Figure 659262DEST_PATH_IMAGE016
the parameter number of the first compression matrix and the parameter number of the second compression matrix are compressed to obtain a first compression matrix and a second compression matrix, and the method specifically comprises the following steps:
will matrix
Figure 678033DEST_PATH_IMAGE019
Setting elements with the values smaller than the compression threshold value corresponding to the row in the vectors of the middle rows as 0, keeping other elements unchanged, obtaining a filter operator Mask matrix, and calculating a first compression matrix and a second compression matrix according to the following formulas:
Figure 824981DEST_PATH_IMAGE020
wherein the content of the first and second substances,
Figure 587401DEST_PATH_IMAGE021
representing a compression matrix, p =1, 2,
Figure 628038DEST_PATH_IMAGE022
representing the multiplication of corresponding elements of two isotypic matrices.
3. The method for mining and parallel processing of mass traffic data knowledge according to claim 1 or 2, wherein in step S17, the compression threshold is updated after each L iterative training, and the value range of L is 1000 to 1500.
CN202110456757.1A 2021-04-27 2021-04-27 Mass traffic data knowledge mining and parallel processing method Active CN113052313B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110456757.1A CN113052313B (en) 2021-04-27 2021-04-27 Mass traffic data knowledge mining and parallel processing method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110456757.1A CN113052313B (en) 2021-04-27 2021-04-27 Mass traffic data knowledge mining and parallel processing method

Publications (2)

Publication Number Publication Date
CN113052313A true CN113052313A (en) 2021-06-29
CN113052313B CN113052313B (en) 2021-10-15

Family

ID=76520620

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110456757.1A Active CN113052313B (en) 2021-04-27 2021-04-27 Mass traffic data knowledge mining and parallel processing method

Country Status (1)

Country Link
CN (1) CN113052313B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113657687A (en) * 2021-08-30 2021-11-16 国家电网有限公司 Power load prediction method based on feature engineering and multi-path deep learning

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150324501A1 (en) * 2012-12-12 2015-11-12 University Of North Dakota Analyzing flight data using predictive models
CN109508812A (en) * 2018-10-09 2019-03-22 南京航空航天大学 A kind of aircraft Trajectory Prediction method based on profound memory network
CN110443411A (en) * 2019-07-16 2019-11-12 青岛民航凯亚系统集成有限公司 Method based on the ADS-B data prediction flight landing time
CN111292563A (en) * 2020-05-12 2020-06-16 北京航空航天大学 Flight track prediction method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20150324501A1 (en) * 2012-12-12 2015-11-12 University Of North Dakota Analyzing flight data using predictive models
CN109508812A (en) * 2018-10-09 2019-03-22 南京航空航天大学 A kind of aircraft Trajectory Prediction method based on profound memory network
CN110443411A (en) * 2019-07-16 2019-11-12 青岛民航凯亚系统集成有限公司 Method based on the ADS-B data prediction flight landing time
CN111292563A (en) * 2020-05-12 2020-06-16 北京航空航天大学 Flight track prediction method

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113657687A (en) * 2021-08-30 2021-11-16 国家电网有限公司 Power load prediction method based on feature engineering and multi-path deep learning
CN113657687B (en) * 2021-08-30 2023-09-29 国家电网有限公司 Power load prediction method based on feature engineering and multipath deep learning

Also Published As

Publication number Publication date
CN113052313B (en) 2021-10-15

Similar Documents

Publication Publication Date Title
Aktürk et al. Aircraft rescheduling with cruise speed control
CN111401601B (en) Delay propagation-oriented flight take-off and landing time prediction method
CN108519988B (en) Aviation delay causal relationship network construction method based on Glangel inspection
CN109190700B (en) Quantitative analysis method for aviation delay propagation
CN116468186B (en) Flight delay time prediction method, electronic equipment and storage medium
CN113052313B (en) Mass traffic data knowledge mining and parallel processing method
Dai et al. Modeling go-around occurrence using principal component logistic regression
Zhang et al. A hierarchical heuristic approach for solving air traffic scheduling and routing problem with a novel air traffic model
CN112270445A (en) Flight delay wave and comprehensive evaluation method based on statistical analysis and classification prediction
Adeke Modelling of queuing process at airport check-in system: a case study of Manchester and Leeds-Bradford airports
CN110796315A (en) Departure flight delay prediction method based on aging information and deep learning
Wieland et al. Predicting sector complexity using machine learning
Guo et al. Deep-Learning-Based Model for Accident-Type Prediction During Approach and Landing
Thiagaraj et al. A queueing model for airport capacity and delay analysis
Xu et al. Statistical analysis of resilience in an air transport network
Provan et al. Optimization models for strategic runway configuration management under weather uncertainty
CN103489040B (en) A kind of flight collision solution desorption method comprising Local Search
Park et al. Situational Anomaly Detection Using Multi-agent Trajectory Prediction for Terminal Airspace Operations
Dönmez Aircraft sequencing under the uncertainty of the runway occupancy times of arrivals during the backtrack procedure
Shortle et al. Uncertainty importance analysis for aviation event trees
Yang et al. Aircraft trajectory prediction and aviation safety in ADS-B failure conditions based on neural network
Stover et al. Data-driven modeling of aircraft midair separation violation
CN117351786B (en) Flight integrated scheduling method under multi-element constraint
Wang et al. The Integrated Evaluation of Airport Ground Movement Performance Based on GA-BP Network
Cai et al. Multiairport Departure Scheduling via Multiagent Reinforcement Learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant