CN116070170A - A method and system for cloud-side-device data fusion processing based on deep learning - Google Patents

Abstract

The invention discloses a cloud edge data fusion processing method and a cloud edge data fusion processing system based on deep learning, which are used for presetting a cloud edge data fusion model; the method comprises the steps of obtaining multi-source heterogeneous data through terminal equipment and processing the multi-source heterogeneous data to obtain various homologous heterogeneous data; training a cloud edge end data fusion model, inputting various homologous heterogeneous data to an edge computing platform in the fusion model, processing to obtain a plurality of advanced fusion feature vectors and a plurality of edge decisions, and feeding back the edge decisions to a terminal device and feeding back to a cloud computing center in the fusion model; inputting the plurality of high-level fusion feature vectors to a cloud computing center, obtaining a cloud decision through cloud computing center model processing, and feeding back the cloud decision to the terminal equipment and the edge computing platform. By establishing a distributed data processing system, the method not only can effectively solve the communication and calculation problems caused by multi-source heterogeneous data, but also can promote the overall decision making and scheduling capability of the system and promote the comprehensive development of the Internet of things.

Description

A method and system for cloud-side-device data fusion processing based on deep learning

technical field

The present invention relates to the field of cloud-side-device data fusion, in particular to a method and system for processing cloud-side-device data fusion based on deep learning.

Background technique

With the advent of the Internet of Everything era, computing demands have exploded. The traditional cloud computing architecture has been unable to meet the massive data computing demands brought about by the explosion of Internet traffic and the geometric growth of data, and the traditional architecture of cloud computing is slowing down. And as the computing and storage capabilities of edge computing platforms and terminal devices continue to improve, it will be an important development trend in the future to sink cloud computing tasks to the edge side and device side, and to create a cloud-edge-device collaborative architecture.

In traditional cloud computing, the construction and operation of cloud computing centers are carried out in the form of overall collection, and customers need to use cloud computing resources through the Internet. With the development of cloud computing technology and the popularization of applications, cloud computing has the following limitations: first, for the situation where the data collected by the front end is too large, the traditional data upload method is costly and inefficient; second, for real-time interaction In the scenario, all data and calculations are concentrated in the computing center, and the cost of information transmission is high, and the delay is high; third, the application requirements for high stability and continuity are strongly dependent on stable cloud services and networks, resulting in robust application requirements and reduced reliability.

Therefore, edge computing has begun to come into the sight of major Internet and cloud service vendors. Edge computing is mainly for data processing at the edge of the network and near the data source. The edge platform integrates computing, storage, transmission and self-management. Its real-time response characteristics will greatly improve the efficiency of data collection and advanced applications.

At present, the cloud-edge-device integrated collaborative computing system is in its infancy. And different from the centralized multi-source heterogeneous data fusion of traditional cloud computing, the distributed multi-source heterogeneous data fusion of cloud-edge-device collaboration will face many challenges. Among them, how to effectively allocate and utilize computing resources at all levels, improve data processing speed and capabilities, effectively train distributed deep learning models, and complete the fusion of multi-source heterogeneous data and information are urgent problems to be solved.

Contents of the invention

The present invention designs a cloud-side-device data fusion processing model and method based on deep learning to complete the efficient processing and effective fusion of multi-source heterogeneous data in the collaborative integration of cloud-side-device.

A deep learning-based cloud-side-end data fusion processing method, the method comprising:

S1. Build a cloud-edge-end data fusion model. The cloud-edge-end data fusion model includes multiple terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, and a cloud computing center model. The edge computing platform model is set on the edge computing platform , the cloud computing center model is set on the cloud computing center, multiple terminal devices are connected to the edge computing platform, the edge computing platform is connected to the cloud computing center, and the cloud computing center is connected to multiple terminal devices;

S2. After using multiple terminal devices to obtain multi-source heterogeneous data, preprocess the multi-source heterogeneous data to obtain various homogeneous and heterogeneous data;

S3. Use a variety of homogeneous and heterogeneous data as the training set to train the cloud-side data fusion model, backpropagate and update parameters until the trained cloud-side data fusion model is obtained, and the cloud-side data fusion model training ends Finally, a variety of homogeneous and heterogeneous data collected and processed by multiple terminal devices are input as data sets into the trained cloud-side-end data fusion model;

S4. After the edge computing platform in the trained cloud-edge-end data fusion model receives a variety of homogeneous and heterogeneous data in the data set, it processes through the edge computing platform model to obtain multiple advanced fusion feature vectors and multiple edge decisions. Transmit multiple advanced fusion feature vectors to the cloud computing center, feed back multiple side decisions to multiple terminal devices, control the collection of multi-source heterogeneous data by multiple terminal devices, and feed forward multiple side decisions to the cloud computing center ;

S5. The cloud computing center in the trained cloud-edge-end data fusion model receives multiple high-level fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decision-making; the cloud computing center controls cloud computing according to multiple feedforward edge decisions Cloud decision-making at the center; the cloud computing center feeds back cloud decision-making to multiple terminal devices, controls the collection of multi-source heterogeneous data by multiple terminal devices, feeds back cloud decision-making to the edge computing platform, and controls the side decision-making of the edge computing platform.

Preferably, after obtaining multi-source heterogeneous data by using multiple terminal devices in S2, the multi-source heterogeneous data is preprocessed to obtain a variety of homogeneous and heterogeneous data. The specific process includes:

S21. Using multiple terminal devices to obtain multi-source heterogeneous data;

S22. Calibrate the multi-source heterogeneous data into multi-source heterogeneous data with synchronous frequency by means of time stamp sampling and linear interpolation;

S23. Standardize the multi-source heterogeneous data with synchronous frequency to obtain multiple homogeneous and heterogeneous data.

Preferably, multiple homologous and heterogeneous data are obtained in S23, and the specific formula is:

Among them, x _ij ' represents the jth homologous heterogeneous data in the i-th homologous heterogeneous data, x _ij represents the j-th homologous heterogeneous data before normalization in the i-th homologous heterogeneous data, j=1,2,...,N, N represents the total number of the i-th homologous heterogeneous data, μ _i and δ _i represent the overall mean and variance of the i-th homologous heterogeneous data.

Preferably, the edge computing platform model includes a plurality of computing node models set in parallel, and after the edge computing platform in the cloud-edge-end data fusion model trained in S4 receives various homologous and heterogeneous data in the data set, it passes through the edge computing platform The model is processed to obtain multiple advanced fusion feature vectors and multiple edge decisions, including:

S41. Store various homologous and heterogeneous data in the data set in multiple computing node models, wherein one computing node model stores a kind of homologous and heterogeneous data;

S42. The computing node model includes a neural network, a first multi-layer perceptron and a first Softmax classifier, and the neural network is used to perform feature extraction on homologous and heterogeneous data stored in the corresponding computing node model to obtain a primary fusion feature vector, The first multi-layer perceptron performs feature-level fusion on the primary fusion feature vector to obtain the high-level fusion feature vector;

S43. The first Softmax classifier is used to classify the high-level fusion feature vector, and obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector;

S44. Calculate the decision probability of the preliminary decision according to the normalization method, and select the preliminary decision with the largest decision probability value as the edge decision corresponding to the output of the calculation node model;

S45. Traversing various homologous and heterogeneous data, repeating steps S42-S44, to obtain multiple high-level fusion feature vectors and multiple edge decisions.

Preferably, in S42, the first multi-layer perceptron performs feature-level fusion on the primary fusion feature vector to obtain the advanced fusion feature vector, and the specific formula is:

in,

In the formula, F _i ' represents the high-level fusion feature vector of the i-th homologous and heterogeneous data, F _i represents the primary fusion feature vector of the i-th homologous and heterogeneous data, W _l represents the l-th learnable feature mapping matrix , 1≤l≤L, L represents the total number of feature mapping matrices, f _ij ^t represents the primary fusion feature output after the t-th cycle of the i-th homologous heterogeneous data, and N represents the i-th homologous heterogeneous total number of data.

Preferably, the cloud computing center in the cloud-side-end data fusion model trained in S5 receives a plurality of advanced fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decision-making, specifically including:

S51. The cloud computing center model includes a converter network, a second multi-layer perceptron and a second Softmax classifier, and the converter network is used to perform same-dimensional encoding and feature fusion on multiple advanced fusion feature vectors, and correspondingly obtain multiple decision-making levels Fusion feature vectors, the second multi-layer perceptron re-integrates multiple decision-level fusion feature vectors to obtain a re-fusion feature vector, and the second Softmax classifier classifies the re-fusion feature vectors to obtain each of the re-fusion feature vectors The final decision corresponding to the feature;

S52. Calculate the decision probability of the final decision according to the normalization method, and select the final decision with the largest decision probability value as the cloud decision.

Preferably, the converter network in S51 is used to perform same-dimensional encoding and feature fusion on multiple advanced fusion feature vectors, and correspondingly obtain multiple decision-level fusion feature vectors. The specific formula is:

In the formula, R _i represents the decision-level fusion feature vector corresponding to the i-th homologous and heterogeneous data, and F _oi ', F _pi ', F _qi 'denotes the same-dimension encoded high-level Fusion feature vector, ReLU represents activation function, d represents feature dimension, W ₁ and W ₂ represent learnable feature mapping matrix, b ₁ and b ₂ represent learnable offset matrix.

Preferably, the edge computing platform model includes a plurality of computing node models arranged in parallel, the edge computing platform includes a plurality of computing nodes arranged in parallel, each computing node includes multiple computing devices, and each computing node corresponds to a computing node model, S3 A variety of homogeneous and heterogeneous data is used as the training set to train the cloud-edge-device data fusion model. Specifically, the edge computing platform model in the cloud-edge-device data fusion model is trained in a data-parallel training method, including:

S31. Store various homologous and heterogeneous data in the training set in multiple computing node models in the edge computing platform model, and each computing node model stores a kind of homologous and heterogeneous data;

S32. For each computing node model in the edge computing platform model, use a variety of homologous and heterogeneous data in the training set for training: each computing node model uses the homologous and heterogeneous data stored separately for model training, when each computing node model When the accuracy of the model is no longer improved, stop updating the parameters of all computing node models;

S33. For multiple computing devices in each computing node corresponding to each computing node model, use homogeneous and heterogeneous data stored in each computing node model for training: use distributed data parallel technology to evenly distribute homogeneous and heterogeneous data For each computing device, load the computing node model corresponding to the computing node on each computing device, and use the method of parameter sharing to update the computing node model parameters on each computing device synchronously and in parallel to ensure the consistency of the computing node models on each computing device When the accuracy of the computing node model on each computing device is no longer improved, stop the training of the current computing node model.

Preferably, S3 uses a variety of homogeneous and heterogeneous data as the training set to train the cloud-side data fusion model, specifically, the cloud-side data fusion model is trained in a model-parallel manner, specifically including:

S34. Distributed model forward reasoning of the cloud-edge-device data fusion model: use various homologous and heterogeneous data in the training set as the input of each computing node model in the edge computing platform model, and after calculation reasoning, when the edge computing platform model When the accuracy no longer changes, the calculation reasoning result of the edge computing platform model is obtained, and the calculation reasoning result of the edge computing platform model is used as the input of the cloud computing center model. After calculation reasoning, when the accuracy of the cloud computing center model no longer changes, Output the calculation and reasoning results of the cloud computing center model. When the calculation and reasoning of the cloud computing center model ends, that is, the forward reasoning process of the cloud-side-device data fusion model ends, the calculation and reasoning results of the cloud-computing center model are the results of the cloud-side-device data fusion model. Calculation of inference results;

S35. Update the distributed model parameters of the cloud-edge-end data fusion model: on the cloud computing center model, use the calculation and reasoning results of the cloud computing center model to calculate the decision-making accuracy, and calculate the accuracy variation to determine whether the accuracy of the cloud computing center model increases, If the accuracy increases, calculate the loss of the cloud computing center model, and reversely calculate the parameter gradient and update the model cloud computing center model parameters. When all the parameters of the cloud computing center model are updated and the accuracy of the cloud computing center model no longer increases , transfer the parameter gradient to the edge computing platform model, and calculate the parameter gradient on each computing node model in the edge computing platform model and update the parameters of each computing node model, when the parameter update of all computing node models in the edge computing platform model is completed , that is, the parameter update of the cloud-edge data fusion model is completed, and the training of the cloud-edge data fusion model is completed.

A cloud-edge-end data fusion processing system based on deep learning, which adopts a cloud-edge-end data fusion processing method based on deep learning to fuse and process cloud-edge-end data. The system includes multiple terminal devices, edge computing platforms, and edge computing platform models , cloud computing center, cloud computing center model, the edge computing platform model is set on the edge computing platform, the cloud computing center model is set on the cloud computing center, multiple terminal devices are connected to the edge computing platform, the edge computing platform and cloud computing The center is connected, and the cloud computing center is connected with multiple terminal devices, among which:

Multiple terminal devices are used to obtain multi-source heterogeneous data, process multi-source heterogeneous data, and obtain various homogeneous and heterogeneous data;

The edge computing platform is used to receive a variety of homogeneous and heterogeneous data, process a variety of homogeneous and heterogeneous data through the edge computing platform model, obtain multiple advanced fusion features and corresponding multiple side decisions, and feed back multiple side decisions For multiple terminal devices, control the collection of homogeneous and heterogeneous data by multiple terminal devices, and feed forward multiple side decisions to the cloud computing center;

The cloud computing center performs feature fusion on multiple high-level fusion features through the cloud computing center model to obtain decision-level fusion features and cloud decision-making; the cloud computing center receives multiple feed-forward edge decisions to control the cloud decision-making of the cloud computing center; the cloud computing center will Cloud decision-making is fed back to multiple terminal devices to control the collection of multi-source heterogeneous data by multiple terminal devices, and cloud decision-making is fed back to the edge computing platform to control the side decision-making of the edge computing platform.

The above-mentioned deep learning-based cloud-side-device data fusion processing method and system can not only effectively solve the communication and computing problems caused by multi-source heterogeneous data, but also create a cloud-side-device collaboration system and a data fusion model based on deep learning. It can also improve the overall decision-making and scheduling capabilities of the system, break the limitations of traditional centralized cloud computing, establish an efficient distributed data processing system, promote the overall development of the Internet of Things, promote the digital transformation of the economy and society, and open a new era of Internet of Everything.

Description of drawings

Fig. 1 is a flow chart of a cloud-side-device data fusion processing method based on deep learning in an embodiment of the present invention;

2 is a schematic structural diagram of a cloud-side-end data fusion processing system based on deep learning in an embodiment of the present invention;

Fig. 3 is a flow chart of training a cloud-edge-device data fusion model based on deep learning in an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings.

A deep learning-based cloud-side-end data fusion processing method, the method comprising:

S1. Build a cloud-edge-end data fusion model. The cloud-edge-end data fusion model includes multiple terminal devices, an edge computing platform, an edge computing platform model, a cloud computing center, and a cloud computing center model. The edge computing platform model is set on the edge computing platform , the cloud computing center model is set on the cloud computing center, multiple terminal devices are connected to the edge computing platform, the edge computing platform is connected to the cloud computing center, and the cloud computing center is connected to multiple terminal devices;

S2. After using multiple terminal devices to obtain multi-source heterogeneous data, preprocess the multi-source heterogeneous data to obtain various homogeneous and heterogeneous data;

S3. Use a variety of homogeneous and heterogeneous data as the training set to train the cloud-side data fusion model, backpropagate and update parameters until the trained cloud-side data fusion model is obtained, and the cloud-side data fusion model training ends Finally, a variety of homogeneous and heterogeneous data collected and processed by multiple terminal devices are input as data sets into the trained cloud-side-end data fusion model;

S4. After the edge computing platform in the trained cloud-edge-end data fusion model receives a variety of homogeneous and heterogeneous data in the data set, it processes through the edge computing platform model to obtain multiple advanced fusion feature vectors and multiple edge decisions. Transmit multiple advanced fusion feature vectors to the cloud computing center, feed back multiple side decisions to multiple terminal devices, control the collection of multi-source heterogeneous data by multiple terminal devices, and feed forward multiple side decisions to the cloud computing center ;

S5. The cloud computing center in the trained cloud-edge-end data fusion model receives multiple high-level fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decision-making; the cloud computing center controls cloud computing according to multiple feedforward edge decisions The cloud decision-making of the center; the cloud computing center feeds back the cloud decision-making to multiple terminal devices, controls the collection of multi-source heterogeneous data by multiple terminal devices, feeds back the cloud decision-making to the edge computing platform, and controls the side decision-making of the edge computing platform.

Specifically, referring to Fig. 1 and Fig. 2, Fig. 1 is a flow chart of a deep learning-based cloud edge data fusion processing method in an embodiment of the present invention, and Fig. 2 is a cloud edge data fusion processing method based on deep learning in an embodiment of the present invention Schematic diagram of the structure of the terminal data fusion processing system.

A cloud-side-end data fusion processing method based on deep learning, including:

1) Build a cloud-edge-end data fusion model: the cloud-edge-end data fusion model includes multiple terminal devices, edge computing platforms, edge computing platform models, cloud computing centers, cloud computing center models, and edge computing platform models are set on edge computing platforms , the cloud computing center model is set on the cloud computing center, and multiple terminal devices are connected to the edge computing platform and the cloud computing center, and the edge computing platform is connected to the cloud computing center;

2) Data acquisition and preprocessing: Use various sensors, mobile phones, personal computers and other terminal devices to acquire multi-source heterogeneous data such as images, videos, sounds, texts, etc., in order to ensure the synchronization between multi-source heterogeneous data, use soft synchronization The way to obtain multi-source heterogeneous data with synchronous frequency. In order to avoid or reduce the influence of the heterogeneity of each homogeneous heterogeneous data in the multi-source heterogeneous data on each computing node model in the edge computing platform model, a standardized method (such as Z-Score) is usually used for preprocessing to Reduce the difference between homologous and heterogeneous data density, probability distribution and correlation of internal attributes, and eliminate the inequalities of homologous and heterogeneous data.

3) Use a variety of homogeneous and heterogeneous data as the training set to train the cloud-side data fusion model, backpropagate and update parameters until the trained cloud-side data fusion model is obtained, and the cloud-side data fusion model training is over Finally, a variety of homogeneous and heterogeneous data collected and processed by multiple terminal devices are input as data sets into the trained cloud-side-end data fusion model;

4) The edge computing platform in the trained cloud-edge-end data fusion model receives a variety of homogeneous and heterogeneous data in the data set, and processes them through the edge computing platform model to obtain multiple advanced fusion feature vectors and multiple edge decisions. The obtained multiple high-level fusion feature vectors are transmitted to the cloud computing center, and the obtained multiple side decisions are fed back to multiple terminal devices to control the collection of multi-source heterogeneous data by multiple terminal devices, and the multiple side decisions are fed forward to the cloud computing center;

5) The cloud computing center in the trained cloud-edge-end data fusion model receives multiple high-level fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decision-making; the cloud computing center controls cloud computing based on multiple feedforward edge decisions The cloud decision-making of the center; the cloud computing center feeds back the cloud decision-making to multiple terminal devices, controls the collection of multi-source heterogeneous data by multiple terminal devices, feeds back the cloud decision-making to the edge computing platform, and controls the edge decision-making in the edge computing platform.

In one embodiment, after multiple terminal devices are used in S2 to obtain multi-source heterogeneous data, the multi-source heterogeneous data is preprocessed to obtain various homogeneous and heterogeneous data. The specific process includes:

S21. Using multiple terminal devices to obtain multi-source heterogeneous data;

S22. Calibrate the multi-source heterogeneous data into multi-source heterogeneous data with synchronous frequency by means of time stamp sampling and linear interpolation;

S23. Standardize the multi-source heterogeneous data with synchronous frequency to obtain multiple homogeneous and heterogeneous data.

In one embodiment, multiple homologous and heterogeneous data are obtained in S23, and the specific formula is:

Among them, x _ij ' represents the jth homologous heterogeneous data in the i-th homologous heterogeneous data, x _ij represents the j-th homologous heterogeneous data before normalization in the i-th homologous heterogeneous data, j=1,2,...,N, N represents the total number of the i-th homologous heterogeneous data, μ _i and δ _i represent the overall mean and variance of the i-th homologous heterogeneous data.

Specifically, first use soft synchronization to acquire multi-source heterogeneous data with the same sampling frequency with time stamps, and calibrate them through time stamp sampling and linear interpolation to obtain synchronous frequency data, so as to eliminate the problems caused by time deviation. influence; then for each type of heterogeneous data in multi-source heterogeneous data, use the Z-Score standardization method to reduce the difference between the density, probability distribution and correlation of internal attributes of homogeneous and heterogeneous data , to eliminate the inequalities of homogeneous and heterogeneous data:

Among them, x _ij ' represents the jth homologous heterogeneous data in the i-th homologous heterogeneous data, x _ij represents the j-th homologous heterogeneous data before normalization in the i-th homologous heterogeneous data, j=1,2,...,N, N represents the total number of the i-th homologous heterogeneous data, μ _i and δ _i represent the overall mean and variance of the i-th homologous heterogeneous data.

In one embodiment, the edge computing platform model includes a plurality of computing node models set in parallel, after the edge computing platform in the cloud-edge-end data fusion model trained in S4 receives various homologous and heterogeneous data in the data set, it passes The edge computing platform model is processed to obtain multiple advanced fusion feature vectors and multiple edge decisions, including:

S41. Store various homologous and heterogeneous data in the data set in multiple computing node models, wherein one computing node model stores a kind of homologous and heterogeneous data;

S42. The computing node model includes a neural network, a first multi-layer perceptron and a first Softmax classifier, and the neural network is used to perform feature extraction on homologous and heterogeneous data stored in the corresponding computing node model to obtain a primary fusion feature vector, The first multi-layer perceptron performs feature-level fusion on the primary fusion feature vector to obtain the high-level fusion feature vector;

S43. The first Softmax classifier is used to classify the high-level fusion feature vector, and obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector;

S44. Calculate the decision probability of the preliminary decision according to the normalization method, and select the preliminary decision with the largest decision probability value as the edge decision corresponding to the output of the calculation node model;

S45. Traversing various homologous and heterogeneous data, repeating steps S42-S44, to obtain multiple high-level fusion feature vectors and multiple edge decisions.

In one embodiment, in S42, the first multi-layer perceptron performs feature-level fusion on the primary fusion feature vector to obtain the high-level fusion feature vector, and the specific formula is:

in,

In the formula, F _i ' represents the high-level fusion feature vector of the i-th homologous and heterogeneous data, F _i represents the primary fusion feature vector of the i-th homologous and heterogeneous data, W _l represents the l-th learnable feature mapping matrix , 1≤l≤L, L represents the total number of feature mapping matrices, f _ij ^t represents the primary fusion feature output after the t-th cycle of the i-th homologous heterogeneous data, and N represents the i-th homologous heterogeneous total number of data.

Specifically, the neural network can be used to extract the features of various homologous and heterogeneous data in the data set: for the one-dimensional homologous and heterogeneous data, the cyclic neural network RNN can be used for one-dimensional feature extraction to obtain one-dimensional features, that is, One-dimensional feature vector; for multi-dimensional homologous and heterogeneous data, convolutional neural network (CNN) can be used for multi-dimensional feature extraction to obtain multi-dimensional features.

Taking the one-dimensional feature extraction of one-dimensional homologous heterogeneous data by using the recurrent neural network RNN as an example, the feature extraction method is as follows:

为第i种第j个同源异构数据第t次循环后输出的特征(x_ij'相当于

在t＝0时刻的同源异构数据)，

为第i种第j个同源异构数据第t-1次循环后输出的特征，X^t表示第t次循环后的输入特征，V为映射矩阵，Relu表示激活函数，t表示第t次循环。in,

is the feature output after the t-th cycle of the j-th homologous heterogeneous data of the i-th type (x _ij ' is equivalent to

Homogeneous data at time t=0),

is the feature output after the t-1th cycle of the i-th homologous heterogeneous data, X ^t represents the input feature after the t-th cycle, V is the mapping matrix, Relu represents the activation function, and t represents the t-th cycle cycle.

The global average pooling method GAP is used to convert multi-dimensional features into one-dimensional feature vectors, and a cascade method is used for feature splicing and preliminary fusion. While ensuring the fusion of homogeneous and heterogeneous data features, it can also solve the problem of heterogeneous data. The problem of different eigenvector lengths due to information density:

表示第i种第j个同源异构数据第t次循环后输出的初级融合特征，即第i种第j个同源异构数据x_ij'第t次循环后输出的特征，N表示不同终端设备提供的同源异构数据的总数，1≤j≤N。Among them, F _i represents the primary fusion feature vector of the i-th homologous and heterogeneous data, and F _i is a 1*N vector,

Indicates the primary fusion feature output after the t-th cycle of the i-th homologous heterogeneous data of the i-th type, that is, the output feature of the j-th homologous heterogeneous data x _ij ' of the i-th type after the t-th cycle, N means different The total number of homogeneous and heterogeneous data provided by the terminal device, 1≤j≤N.

The multi-layer perceptron MLP is used to perform feature-level fusion on the primary fusion feature vectors of each homologous and heterogeneous data, and the advanced fusion feature vectors are obtained:

表示第i种第j个同源异构数据第t次循环后输出的高级融合特征，M_il表示第i种同源异构数据第l个可学习的特征映射矩阵，l＝1,2,...,L，L表示特征映射矩阵的总数，代表使用L层的MLP。Among them, F _i 'represents the high-level fusion feature vector of the i-th homologous and heterogeneous data, and F _i 'is also a 1*N vector,

Indicates the advanced fusion feature output after the t-th cycle of the i-th homologous heterogeneous data of the i-th type, M _il represents the l-th learnable feature mapping matrix of the i-th homologous heterogeneous data, l=1,2, ..., L, where L represents the total number of feature map matrices, representing an MLP using L layers.

The high-level fusion feature vector is classified by the first Softmax classifier to obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector.

Calculate the decision probability of the preliminary decision by the normalization method:

对应的边决策的决策概率值，0≤S_ij≤1。In the formula, S _ij is the advanced fusion feature of the i-th type j-th homologous heterogeneous data

The decision probability value of the corresponding edge decision, 0≤S _ij ≤1.

Select the preliminary decision with the largest decision probability value as the side decision of the i-th computing node model, that is, the side decision corresponding to the i-th kind of homogeneous and heterogeneous data, and feed back the side decision to multiple different terminal devices. The type device terminal will prompt a pop-up window to remind the user to upload the corresponding data, and issue instructions to the active collection type device terminal to control data collection; upload the i-th advanced fusion feature vector F _i ' and the edge decision of the i-th computing node model to cloud computing center.

In one embodiment, the cloud computing center in the cloud-edge-end data fusion model trained in S5 receives a plurality of advanced fusion feature vectors, processes them through the cloud computing center model, and obtains cloud decision-making, specifically including:

S51. The cloud computing center model includes a converter network, a second multi-layer perceptron and a second Softmax classifier, and the converter network is used to perform same-dimensional encoding and feature fusion on multiple advanced fusion feature vectors, and correspondingly obtain multiple decision-making levels Fusion feature vectors, the second multi-layer perceptron re-integrates multiple decision-level fusion feature vectors to obtain a re-fusion feature vector, and the second Softmax classifier classifies the re-fusion feature vectors to obtain each of the re-fusion feature vectors The final decision corresponding to the feature;

S52. Calculate the decision probability of the final decision according to the normalization method, and select the final decision with the largest decision probability value as the cloud decision.

In one embodiment, the converter network in S51 is used to perform same-dimensional encoding and feature fusion on multiple advanced fusion feature vectors, and correspondingly obtain multiple decision-level fusion feature vectors. The specific formula is:

In the formula, R _i represents the decision-level fusion feature vector corresponding to the i-th homologous and heterogeneous data, and F _oi ', F _pi ', F _qi 'denotes the same-dimension encoded high-level Fusion feature vector, ReLU represents activation function, d represents feature dimension, W ₁ and W ₂ represent learnable feature mapping matrix, b ₁ and b ₂ represent learnable offset matrix.

Specifically, the advanced fusion feature vector provided by each computing node model in the edge computing platform model is firstly one-hot encoded, and transformed into sparse data with only one bit of activation, which can effectively solve the problem of difficult classification of decision-making between multi-source data , and effectively expand the features; then use the PCA dimension reduction operation to reduce the data dimension to alleviate the dimension disaster, and can effectively reduce data noise and ensure feature independence, help the subsequent neural network to learn and extract features, and complete multiple decision probabilities The features are encoded in the same dimension separately. Use the self-attention mechanism in the transformer network Transformer to obtain the correlation between homogeneous and heterogeneous data, and perform decision-level fusion on homologous and heterogeneous data to obtain the decision-level fusion feature vector corresponding to homologous and heterogeneous data:

In the formula, R _i represents the decision-level fusion feature corresponding to the i-th homologous heterogeneous data, ReLU represents the activation function, which is used to introduce nonlinear factors and improve the expressive ability of the neural network, F _oi ', F _pi ', F _qi 'indicates the high-level fusion feature vector encoded in the same dimension corresponding to the i-th homologous and heterogeneous data, d indicates the feature dimension, which is used to avoid the problem of gradient explosion caused by large values after multiple multiplication operations, W ₁ , W ₂ represents a learnable feature mapping matrix, and b ₁ and b ₂ represent learnable offset matrices.

Use the second multi-layer perceptron MLP to re-fuse multiple decision-level fusion feature vectors to obtain a re-fusion feature vector to reduce the dimension of the fusion features, and then use the second Softmax classifier to classify the re-fusion feature vectors to obtain a re-fusion feature vector The final decision corresponding to each feature in the fusion feature vector is converted into a probability distribution ranging from 0 to 1 and 1 according to the normalization method. The specific calculation method is the same as the calculation of multiple preliminary decisions based on the normalization method The decision probability of each preliminary decision in , will not be repeated here. Then, according to the probability distribution, the final decision with the highest decision probability value is selected as the cloud decision, and the cloud decision is fed back to multiple terminal devices to carry out general control of large-scale terminal equipment and control its collection of multi-source heterogeneous data; The decision is fed back to the edge computing platform to control the upload of homogeneous and heterogeneous data.

In one embodiment, the edge computing platform model includes a plurality of computing node models arranged in parallel, the edge computing platform includes a plurality of computing nodes arranged in parallel, each computing node includes a plurality of computing devices, and each computing node corresponds to a computing node Model, S3 uses a variety of homogeneous and heterogeneous data as the training set to train the cloud-edge-device data fusion model. Specifically, the edge computing platform model in the cloud-edge-device data fusion model is trained using a data parallel training method. Specifically include:

S31. Store various homologous and heterogeneous data in the training set in multiple computing node models in the edge computing platform model, and each computing node model stores a kind of homologous and heterogeneous data;

S32. For each computing node model in the edge computing platform model, use a variety of homologous and heterogeneous data in the training set for training: each computing node model uses the homologous and heterogeneous data stored separately for model training, when each computing node model When the accuracy of the model is no longer improved, stop updating the parameters of all computing node models;

S33. For multiple computing devices in each computing node corresponding to each computing node model, use homogeneous and heterogeneous data stored in each computing node model for training: use distributed data parallel technology to evenly distribute homogeneous and heterogeneous data For each computing device, load the computing node model corresponding to the computing node on each computing device, and use the method of parameter sharing to update the computing node model parameters on each computing device synchronously and in parallel to ensure the consistency of the computing node model on each computing device When the accuracy of the computing node model on each computing device is no longer improved, stop the training of the current computing node model.

Specifically, the multi-source heterogeneous data obtained by multiple different terminal devices is decoupled into a variety of homogeneous heterogeneous data, which are distributed and stored in each computing node model in the edge computing platform model, and each computing node model stores a A variety of homogeneous and heterogeneous data, which is conducive to reducing data access and transmission; the edge computing platform model uses a variety of homologous and heterogeneous data parallel training: each computing node model uses the homologous and heterogeneous data stored in each node model for training , and there is no mutual communication and data transmission between nodes. When the accuracy of each node model is no longer improved, the parameter update of all node models is stopped; in the single computing node model in the edge computing platform model, the corresponding homologous heterogeneous Data parallel training: Use distributed data parallel DDP technology to synchronously distribute training data to each computing device in the computing node, and use parameter sharing to update the parameters of the node model synchronously and in parallel. When the accuracy of the computing node model does not improve stop training.

The DDP technology is as follows:

1) Start multiple processes to be responsible for multiple computing devices in multiple computing nodes. Each process loads the computing node model of the corresponding node on each computing device, and allocates training data and training data according to the performance of each computing device in the node. Task.

2) After each computing node model performs a forward inference process, the calculation result of each computing node model is transmitted to each computing device in the node, the result is analyzed and the training loss is calculated, and the training loss is calculated after completion Pass it to the computing node model, and perform gradient calculation and model parameter update for each computing device in the computing node. When all computing device parameters are updated, the current computing node model completes a model parameter update.

3) Since the model structure and initialization parameters on multiple computing devices in each computing node are the same, and the same training loss and training strategy are used for gradient calculation and parameter update, after the training, the multiple computing devices in each computing node model The model parameters are exactly the same on each computing device.

In one embodiment, S3 uses various homogeneous and heterogeneous data as the training set to train the cloud-side data fusion model, specifically, the cloud-side data fusion model is trained in a model-parallel manner, specifically including:

S34. Distributed model forward reasoning of the cloud-edge-device data fusion model: use various homologous and heterogeneous data in the training set as the input of each computing node model in the edge computing platform model, and after calculation reasoning, when the edge computing platform model When the accuracy no longer changes, the calculation reasoning result of the edge computing platform model is obtained, and the calculation reasoning result of the edge computing platform model is used as the input of the cloud computing center model. After calculation reasoning, when the accuracy of the cloud computing center model no longer changes, Output the calculation and reasoning results of the cloud computing center model. When the calculation and reasoning of the cloud computing center model ends, that is, the forward reasoning process of the cloud-side-device data fusion model ends, the calculation and reasoning results of the cloud-computing center model are the results of the cloud-side-device data fusion model. Calculation of inference results;

S35. Update the distributed model parameters of the cloud-edge-end data fusion model: on the cloud computing center model, use the calculation and reasoning results of the cloud computing center model to calculate the decision-making accuracy, and calculate the accuracy variation to determine whether the accuracy of the cloud computing center model increases, If the accuracy increases, calculate the loss of the cloud computing center model, and reversely calculate the parameter gradient and update the model cloud computing center model parameters. When all the parameters of the cloud computing center model are updated and the accuracy of the cloud computing center model no longer increases , transfer the parameter gradient to the edge computing platform model, and calculate the parameter gradient on each computing node model in the edge computing platform model and update the parameters of each computing node model, when the parameter update of all computing node models in the edge computing platform model is completed , that is, the parameter update of the cloud-edge data fusion model is completed, and the training of the cloud-edge data fusion model is completed.

Specifically, referring to FIG. 3 , FIG. 3 is a flowchart of training a cloud-edge-device data fusion model based on deep learning in an embodiment of the present invention.

Through the model parallel training method, a variety of homogeneous and heterogeneous data are used to fine-tune the parameters of the edge computing platform model and the cloud computing center model, and complete the training of the cloud edge data fusion model:

1) Distributed model forward reasoning of the cloud-edge-device data fusion model: the edge computing platform model uses homogeneous and heterogeneous data as the input of each computing node model, and calculates the model results, decision-making accuracy, and accuracy variation. Judgment, when the accuracy change is greater than zero, it is necessary to calculate the model loss and parameter gradient, and update the model parameters, and use the same source and heterogeneous data as the input of each computing node model again, and perform the above calculation reasoning (calculation model results, Decision-making accuracy and accuracy variation are judged according to the accuracy variation), until the accuracy of the computing node model no longer changes, the computing reasoning of the computing node model ends. After the calculation and reasoning of all computing node models is completed, the calculation and reasoning results of the edge computing platform model are transmitted to the cloud computing center model as input, and the calculation and reasoning are performed (the specific calculation and reasoning method is the same as the calculation and reasoning method of each computing node model above, No more details here), when the calculation and reasoning of the cloud computing center model ends, that is, the forward reasoning process of the cloud-side data fusion model ends. The formula expression of the forward calculation reasoning process is as follows:

y＝(U ₁ (x ₁₁ ')+U ₁ (x ₁₂ ')+···+U ₁ (x _1j '))+···+(U _i (x _i1 ')+U _i (x _i2 ')+···+U _i (x _ij '))

Among them, y represents the final calculation result of the cloud-edge data fusion model, Y represents the cloud computing center model, U _i represents the i-th computing node model in the edge computing platform model, u _i represents the i-th computing node model in the edge computing platform model The calculation result of the node model, that is, the calculation result of the i-th homologous heterogeneous data, x _ij ' represents the input of the i-th computing node model, that is, the j-th homologous heterogeneous data in the i-th homologous heterogeneous data Data, what needs to be explained here is that, assuming that N types of multi-source heterogeneous data are obtained through N different terminal devices, after processing, no more than N types of homogeneous and heterogeneous data are obtained. At this time, the calculation in the edge computing platform model The number of nodes only needs to satisfy "each computing node processes a kind of homogeneous and heterogeneous data", not necessarily N.

2) Update the distributed model parameters of the cloud-side-end data fusion model: on the cloud-side-end data fusion model, use the reasoning results of the cloud-side-end data fusion model to calculate the decision-making accuracy, and calculate the accuracy variation, and judge whether the accuracy of the cloud-side-end data fusion model has increased , if the accuracy increases, calculate the loss of the cloud computing center model, and reversely calculate the parameter gradient and update the cloud computing center model parameters. When all the parameters of the cloud computing center model are updated, and the accuracy of the cloud computing center model will not increase When , pass the parameter gradient to the edge computing platform model, and calculate the parameter gradient and update the model parameters on each computing node model. The parameter update of the model is completed, and the training of the cloud-side data fusion model is completed. The formula for calculating the parameter gradient process is as follows:

Among them, Y' represents the parameter gradient of the cloud computing center model, y represents the final calculation result of the cloud-edge data fusion model, U _i ' represents the parameter gradient of the i-th computing node model in the edge computing platform model, and u _i represents the edge computing The calculation result of the i-th computing node model in the platform model, x _ij ' represents the input of the i-th computing node model, that is, the j-th homologous and heterogeneous data in the i-th homologous and heterogeneous data.

In one embodiment, the cloud computing center includes multiple parallel computing devices, and the cloud computing center model is trained by using the calculation and reasoning results of the edge computing platform model, specifically including:

1) Utilize the inference results of each computing node model in the edge computing platform model to construct the training data of the cloud computing center model;

2) Using DDP technology, the training data is synchronously distributed to each computing device in the cloud computing center, and the model parameters are updated synchronously and in parallel by means of parameter sharing, and the training is stopped when the accuracy of the cloud computing center model no longer improves.

In one embodiment, a deep learning-based cloud-side-device data fusion processing system adopts a deep-learning-based cloud-side-device data fusion processing method to perform fusion processing on cloud-side-device data. The system includes multiple terminal devices, edge computing Platform, edge computing platform model, cloud computing center, cloud computing center model, the edge computing platform model is set on the edge computing platform, the cloud computing center model is set on the cloud computing center, multiple terminal devices are connected to the edge computing platform, The edge computing platform is connected to the cloud computing center, and the cloud computing center is connected to multiple terminal devices, among which:

Multiple terminal devices are used to obtain multi-source heterogeneous data, process multi-source heterogeneous data, and obtain various homogeneous and heterogeneous data;

The edge computing platform is used to receive a variety of homogeneous and heterogeneous data, process a variety of homogeneous and heterogeneous data through the edge computing platform model, obtain multiple advanced fusion features and corresponding multiple side decisions, and feed back multiple side decisions For multiple terminal devices, control the collection of homogeneous and heterogeneous data by multiple terminal devices, and feed forward multiple side decisions to the cloud computing center;

The cloud computing center performs feature fusion on multiple high-level fusion features through the cloud computing center model to obtain decision-level fusion features and cloud decision-making; the cloud computing center receives multiple feed-forward edge decisions to control the cloud decision-making of the cloud computing center; the cloud computing center will Cloud decision-making is fed back to multiple terminal devices to control the collection of multi-source heterogeneous data by multiple terminal devices, and cloud decision-making is fed back to the edge computing platform to control the side decision-making of the edge computing platform.

For the specific limitations of the cloud-side-device data fusion processing system based on deep learning, please refer to the above-mentioned definition of the cloud-side-device data fusion processing method based on deep learning, and will not be repeated here.

The above-mentioned deep learning-based cloud-side-device data fusion processing method and system can not only effectively solve the communication and computing problems caused by multi-source heterogeneous data, but also create a cloud-side-device collaboration system and a data fusion model based on deep learning. It can also improve the overall decision-making and scheduling capabilities of the system, break the limitations of traditional centralized cloud computing, establish an efficient distributed data processing system, promote the overall development of the Internet of Things, promote the digital transformation of the economy and society, and open a new era of Internet of Everything.

The above is a detailed introduction of the deep learning-based cloud-edge-device data fusion processing method and system provided by the present invention. In this paper, specific examples are used to illustrate the principles and implementation modes of the present invention, and the descriptions of the above embodiments are only used to help understand the core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (10)

1. The cloud edge data fusion processing method based on deep learning is characterized by comprising the following steps of:

S1, a cloud edge data fusion model is built, the cloud edge data fusion model comprises a plurality of terminal devices, an edge computing platform model, a cloud computing center and a cloud computing center model, the edge computing platform model is arranged on the edge computing platform, the cloud computing center model is arranged on the cloud computing center, a plurality of terminal devices are connected with the edge computing platform, the edge computing platform is connected with the cloud computing center, and the cloud computing center is connected with a plurality of terminal devices;

s2, preprocessing the multi-source heterogeneous data after the multi-source heterogeneous data are acquired by utilizing a plurality of terminal devices to obtain a plurality of kinds of homologous heterogeneous data;

s3, training the cloud edge data fusion model by using a plurality of homologous heterogeneous data as a training set, and reversely propagating and updating parameters until a trained cloud edge data fusion model is obtained, and after the training of the cloud edge data fusion model is finished, inputting a plurality of homologous heterogeneous data acquired and processed by a plurality of terminal devices into the trained cloud edge data fusion model as a data set;

S4, after the edge computing platform in the trained cloud edge data fusion model receives various homologous heterogeneous data in the data set, processing the data set through the edge computing platform model to obtain a plurality of high-level fusion feature vectors and a plurality of edge decisions, transmitting the high-level fusion feature vectors to the cloud computing center, feeding back the edge decisions to the terminal devices, controlling the terminal devices to acquire the multi-source heterogeneous data, and feeding the edge decisions to the cloud computing center;

s5, a cloud computing center in the trained cloud side data fusion model receives a plurality of high-level fusion feature vectors, and the cloud computing center model is used for processing the high-level fusion feature vectors to obtain a cloud decision; the cloud computing center controls cloud decisions of the cloud computing center according to the feedforward edge decisions; the cloud computing center feeds back the cloud decision to a plurality of terminal devices, controls the acquisition of multi-source heterogeneous data by the plurality of terminal devices, feeds back the cloud decision to the edge computing platform, and controls the edge decision of the edge computing platform.

2. The cloud end data fusion processing method based on deep learning as claimed in claim 1, wherein after the multi-source heterogeneous data is obtained by using a plurality of terminal devices in the step S2, the multi-source heterogeneous data is preprocessed to obtain a plurality of kinds of homologous heterogeneous data, and the specific process includes:

S21, acquiring multi-source heterogeneous data by using a plurality of terminal devices;

s22, calibrating the multi-source heterogeneous data into multi-source heterogeneous data with synchronous frequency in a time stamp sampling and linear interpolation mode;

s23, carrying out standardization processing on the multi-source heterogeneous data with the synchronous frequency to obtain various homologous heterogeneous data.

3. The cloud end data fusion processing method based on deep learning as claimed in claim 2, wherein the step S23 is characterized in that a plurality of homologous heterogeneous data are obtained by a specific formula:

wherein ,x_ij ' represents the j-th homologous heterogeneous data in the i-th homologous heterogeneous data, x _ij Represents the j-th pre-normalization homolog isomer data in the i-th homolog isomer data, j=1, 2,.. _i 、δ _i Mean and variance of the ensemble of the ith homologous heterogeneous data are shown.

4. The method for processing cloud edge data fusion based on deep learning according to claim 3, wherein the edge computing platform model comprises a plurality of computing node models arranged in parallel, and the edge computing platform in the trained cloud edge data fusion model in S4 receives the plurality of homologous heterogeneous data in the data set and then processes the plurality of homologous heterogeneous data through the edge computing platform model to obtain a plurality of high-level fusion feature vectors and a plurality of edge decisions, and the method specifically comprises:

S41, storing multiple homologous heterogeneous data in the data set in multiple computing node models, wherein one computing node model stores one type of homologous heterogeneous data;

s42, the computing node model comprises a neural network, a first multi-layer perceptron and a first Softmax classifier, wherein the neural network is used for extracting features of homologous heterogeneous data stored in the corresponding computing node model to obtain primary fusion feature vectors, and the first multi-layer perceptron performs feature level fusion on the primary fusion feature vectors to obtain advanced fusion feature vectors;

s43, the first Softmax classifier is used for classifying the high-level fusion feature vector to obtain a preliminary decision corresponding to each feature in the high-level fusion feature vector;

s44, calculating the decision probability of the preliminary decision according to a normalization method, and selecting the preliminary decision with the maximum decision probability value as an edge decision output by a corresponding calculation node model;

s45, traversing a plurality of homologous heterogeneous data, and repeating the steps S42-S44 to obtain a plurality of high-level fusion feature vectors and a plurality of edge decisions.

5. The method for processing cloud end data fusion based on deep learning as claimed in claim 4, wherein in the step S42, the first multi-layer perceptron performs feature level fusion on the primary fusion feature vector to obtain an advanced fusion feature vector, and the specific formula is as follows:

wherein ,

in the formula ,F_i ' high-level fusion eigenvector representing ith homologous heterogeneous data, F _i Primary fusion feature vector representing ith homologous heterogeneous data, W _l Represents the first learnable feature mapping matrix, L is more than or equal to 1 and less than or equal to L, L represents the total number of the feature mapping matrices, and f _ij ^t Represents the primary fusion characteristics of the ith and jth homologous heterogeneous data output after the jth cycle, and N represents the total number of the ith homologous heterogeneous data.

6. The deep learning-based cloud edge data fusion processing method of claim 5, wherein a cloud computing center in the trained cloud edge data fusion model in S5 receives a plurality of the advanced fusion feature vectors, and processes the advanced fusion feature vectors through a cloud computing center model to obtain cloud decisions, and the method specifically comprises the following steps:

s51, the cloud computing center model comprises a converter network, a second multi-layer perceptron and a second Softmax classifier, wherein the converter network is used for carrying out same-dimensional coding and feature fusion on a plurality of high-level fusion feature vectors to correspondingly obtain a plurality of decision-level fusion feature vectors, the second multi-layer perceptron is used for carrying out rebusding on the plurality of decision-level fusion feature vectors to obtain a rebusding feature vector, and the second Softmax classifier is used for classifying the rebusding feature vector to obtain a final decision corresponding to each feature in the rebusding feature vector;

S52, calculating the decision probability of the final decision according to a normalization method, and selecting the final decision with the maximum decision probability value as a cloud decision.

7. The method for processing cloud-edge data fusion based on deep learning as claimed in claim 6, wherein in the step S51, the transformer network is configured to perform co-dimensional coding and feature fusion on a plurality of the advanced fusion feature vectors, so as to obtain a plurality of decision-level fusion feature vectors correspondingly, and the specific formula is as follows:

in the formula ,R_i Representing decision-level fusion eigenvectors corresponding to the ith homologous heterogeneous data, F _oi '、F _pi '、F _qi ' represents the same-dimensional coded high-level fusion feature vector corresponding to the ith homologous heterogeneous data, reLU represents an activation function, d represents feature dimension, W ₁ 、W ₂ Representing a learnable feature mapping matrix, b ₁ 、b ₂ Representing a learnable offset matrix.

8. The method for processing cloud-edge data fusion based on deep learning according to claim 1, wherein the edge computing platform model comprises a plurality of computing node models arranged in parallel, the edge computing platform comprises a plurality of computing nodes arranged in parallel, each computing node comprises a plurality of computing devices, each computing node corresponds to one computing node model, the cloud-edge data fusion model is trained by adopting a plurality of homologous heterogeneous data as training sets in S3, and in particular, the training mode of data parallelism is adopted for training the edge computing platform model in the cloud-edge data fusion model, and the method specifically comprises the following steps:

S31, storing various homologous heterogeneous data in the training set in a plurality of computing node models in the edge computing platform model, wherein each computing node model stores one type of homologous heterogeneous data;

s32, training by adopting various homologous heterogeneous data in the training set aiming at each computing node model in the edge computing platform model: each computing node model carries out model training by using the stored homologous heterogeneous data, and when the precision of each computing node model is not improved any more, the parameter updating of all the computing node models is stopped;

s33, training a plurality of computing devices in each computing node corresponding to each computing node model by using homologous heterogeneous data stored in each computing node model: and uniformly distributing the homologous heterogeneous data to each computing device by using a distributed data parallel technology, loading a computing node model corresponding to the computing node on each computing device, synchronously and parallelly updating the computing node model parameters on each computing device by adopting a parameter sharing mode, ensuring the consistency of the computing node models on each computing device, and stopping the training of the current computing node model when the precision of the computing node model on each computing device is not improved.

9. The method for processing the cloud-edge data fusion based on deep learning according to claim 8, wherein the training of the cloud-edge data fusion model by using a plurality of homologous heterogeneous data as training sets in the step S3 specifically comprises the steps of:

s34, forward reasoning of a distributed model of the cloud side data fusion model: taking various homologous heterogeneous data in the training set as the input of each calculation node model in the edge calculation platform model, obtaining a calculation reasoning result of the edge calculation platform model when the precision of the edge calculation platform model is not changed any more through calculation reasoning, taking the calculation reasoning result of the edge calculation platform model as the input of the cloud calculation center model, outputting the calculation reasoning result of the cloud calculation center model when the precision of the cloud calculation center model is not changed any more through calculation reasoning, and ending the forward reasoning process of the cloud edge data fusion model when the calculation reasoning of the cloud calculation center model is ended, wherein the calculation reasoning result of the cloud calculation center model is the calculation reasoning result of the cloud edge data fusion model;

S35, updating distributed model parameters of the cloud side data fusion model: and on the cloud computing center model, calculating decision accuracy by using a calculation reasoning result of the cloud computing center model, calculating an accuracy variation, judging whether the accuracy of the cloud computing center model is increased, if so, calculating the loss of the cloud computing center model, reversely calculating parameter gradients and updating model cloud computing center model parameters, when all the parameters of the cloud computing center model are updated, and when the accuracy of the cloud computing center model is not increased any more, transmitting the parameter gradients to the edge computing platform model, calculating the parameter gradients and updating the parameters of each computing node model on each computing node model in the edge computing platform model, and when the parameter updating of all the computing node models in the edge computing platform model is ended, namely, the parameter updating of the cloud edge data fusion model is ended, and the training of the cloud edge data fusion model is ended.

10. The cloud edge data fusion processing system based on deep learning, which is characterized by comprising a plurality of terminal devices, an edge computing platform model, a cloud computing center and a cloud computing center model, wherein the edge computing platform model is arranged on the edge computing platform, the cloud computing center model is arranged on the cloud computing center, a plurality of terminal devices are connected with the edge computing platform, the edge computing platform is connected with the cloud computing center, and the cloud computing center is connected with a plurality of terminal devices, wherein:

The terminal devices are used for acquiring multi-source heterogeneous data, and processing the multi-source heterogeneous data to obtain various homologous heterogeneous data;

the edge computing platform is used for receiving various homologous heterogeneous data, processing the various homologous heterogeneous data through the edge computing platform model to obtain a plurality of advanced fusion characteristics and a plurality of corresponding edge decisions, feeding the edge decisions back to the terminal devices, controlling the terminal devices to acquire the homologous heterogeneous data, and feeding the edge decisions back to the cloud computing center;

the cloud computing center performs feature fusion on the plurality of advanced fusion features through the cloud computing center model to obtain decision-level fusion features and cloud decisions; the cloud computing center receives a plurality of feedforward edge decisions to control cloud decisions of the cloud computing center; the cloud computing center feeds back the cloud decision to a plurality of terminal devices, controls the acquisition of multi-source heterogeneous data by the plurality of terminal devices, feeds back the cloud decision to the edge computing platform, and controls the edge decision of the edge computing platform.

Images