CN114143833A - Data slice transmission method - Google Patents
Data slice transmission method Download PDFInfo
- Publication number
- CN114143833A CN114143833A CN202111270679.2A CN202111270679A CN114143833A CN 114143833 A CN114143833 A CN 114143833A CN 202111270679 A CN202111270679 A CN 202111270679A CN 114143833 A CN114143833 A CN 114143833A
- Authority
- CN
- China
- Prior art keywords
- data
- slice
- mode
- transmission
- length
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000005540 biological transmission Effects 0.000 title claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000007405 data analysis Methods 0.000 claims abstract description 4
- 238000013500 data storage Methods 0.000 claims abstract description 4
- 238000004458 analytical method Methods 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 230000008034 disappearance Effects 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 238000011160 research Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 16
- 210000002569 neuron Anatomy 0.000 description 8
- 230000000694 effects Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000013145 classification model Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000002364 input neuron Anatomy 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W28/00—Network traffic management; Network resource management
- H04W28/02—Traffic management, e.g. flow control or congestion control
- H04W28/06—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information
- H04W28/065—Optimizing the usage of the radio link, e.g. header compression, information sizing, discarding information using assembly or disassembly of packets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The application provides a data slice transmission method, which comprises the following steps: after data to be sliced are obtained, reading the size and the type of a file; and then automatically selecting a slicing method according to the data attribute information, such as the number of slices and the size of each slice, then selecting the existing available network link, such as wifi, 5G and the like, to transmit data, and finally restoring the data on the slices to the original data at the receiving end. The invention learns the slice model and the link selection model through the LSTM algorithm, further optimizes the slice mode and the transmission link mode, can further improve the efficiency and the accuracy of data processing and transmission on the basis of the existing data processing and transmission method, and provides convenience for subsequent data storage and analysis.
Description
Technical Field
The application relates to the technical field of information data processing and transmission, and mainly relates to a data slice transmission method.
Background
With the rapid development of big data and communication technology, at present, various data in the power industry are processed by adopting a scattered acquisition centralized processing mode gradually, so that under the condition that a large amount of power data needs to be transmitted and corrected in time, the accuracy and safety of data transmission are greatly determined by the processing and transmission of various coupled, batched and continuous data.
Disclosure of Invention
The application aims to provide a data slice transmission method which is used for solving the problems of inflexible scheme selection and low efficiency in traditional data transmission.
In order to achieve the above object, the present application provides the following technical solutions:
the application provides a data slice transmission method, which comprises the following steps:
a method for transmitting a data slice, the method comprising:
the data from the power system is acquired, and the data can be acquired through a Bluetooth mode, a WiFi mode, a 4G/5G mode, a local area network mode, a mobile storage device mode and the like, and the data type, the data size and the data length are preliminarily analyzed, and enough space is prepared for analyzing and processing the data.
And reading the attribute of the power data, classifying the power data according to the size and the length of the data, and determining a pre-slicing model of a data slicing mode.
Slice properties are determined. The LSTM model is used for analyzing the attributes of the data, a slice optimization mode is selected according to the analysis result and the data length, the slice length and the number of slices are adjusted, and the purposes of dividing the original data, packaging the slices and the like in a balanced and rapid mode are achieved as far as possible.
The LSTM model is applied to solving the problems of identification and analysis of the electric power data attributes, data attribute characteristic quantities of the same type are memorized, the internal relation of input mapping to output is established, and reliable identification of the data attributes is achieved.
Compared with a shallow layer identification algorithm, the LSTM algorithm solves the problem of low identification accuracy rate caused by various attribute types; compared with the RNN model, the method can effectively solve the phenomenon of gradient disappearance in the large sample data training process in the past, can improve the reliability of attribute analysis, and provides basic support for subsequent data slicing and transmission.
The slice attributes include: slice pointer, slice length, slice volume. After the LSTM model performs data attribute analysis, the system will select a slice length and method appropriate for the data, slice the data, and edit the slice's pointers, length and volume, along with the slice contents, to synthesize a packet for network link transmission. For transmission.
According to the calculation result of the algorithm, for the length and the transmission complexity of the data packet, in the transmission mode (including but not limited to the modes of Bluetooth, WiFi, 4G/5G, local area network, mobile storage equipment and the like) with good network connection of the existing system, the type of the transmitted link is selected by combining the data receiving mode of the receiving end equipment, and especially for the link of the large-slice data selection network to transmit the slice data
And positioning and extracting a slice data packet according to the data slice method and the pointer address, and restoring slice content data through slice attributes for subsequent data storage and analysis research.
All the processes do not need human participation, the system can be checked before operation and normally operated, and the operation is convenient and quick.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts. The foregoing and other objects, features and advantages of the application will be apparent from the accompanying drawings. Like reference numerals refer to like parts throughout the drawings. The drawings are not intended to be to scale as practical, emphasis instead being placed upon illustrating the subject matter of the present application.
FIG. 1 is a schematic flow diagram of the method of the present application;
FIG. 2 is a schematic diagram of the technical application architecture of the method of the present application;
fig. 3 is a flow chart of an LSTM implementation of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art without any inventive work based on the embodiments in the present application are within the scope of protection of the present application. Thus, the following detailed description of the embodiments of the present application, as presented in the figures, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments obtained by a person of ordinary skill in the art without any inventive work based on the embodiments in the present application are within the scope of protection of the present application.
Examples
As shown in fig. 1, the present application provides a data slice transmission method, and a system architecture of the method is shown in fig. 2, and the specific method includes:
the data from the power system is acquired, and the data can be acquired through a Bluetooth mode, a WiFi mode, a 4G/5G mode, a local area network mode, a mobile storage device mode and the like, and the data type, the data size and the data length are preliminarily analyzed, and enough space is prepared for analyzing and processing the data.
And reading the attribute of the power data, classifying the power data according to the size and the length of the data, and determining a pre-slicing model of a data slicing mode.
Slice properties are determined. The LSTM model is used for analyzing the attributes of the data, a slice optimization mode is selected according to the analysis result and the data length, the slice length and the number of slices are adjusted, and the purposes of dividing the original data, packaging the slices and the like in a balanced and rapid mode are achieved as far as possible.
The LSTM model is applied to solving the problems of identification and analysis of the electric power data attributes, data attribute characteristic quantities of the same type are memorized, the internal relation of input mapping to output is established, and reliable identification of the data attributes is achieved.
Compared with a shallow layer identification algorithm, the LSTM algorithm solves the problem of low identification accuracy rate caused by various attribute types; compared with the RNN model, the method can effectively solve the phenomenon of gradient disappearance in the large sample data training process in the past, can improve the reliability of attribute analysis, and provides basic support for subsequent data slicing and transmission.
The LSTM deep learning algorithm learns underlying laws and characteristics by training a large number of data samples, the entire framework of which is shown in fig. 2. In order to obtain a better learning sample, a data attribute detection and analysis mechanism is introduced into the power data sample to clarify the types and cutting requirements of different power data, and then training, learning and other operations are performed through the LSTM.
The flow of the power data attribute analysis algorithm combined with the LSTM network is mainly divided into the following steps, as shown in fig. 3.
(1) Preparing training and test sample data
When a training sample is prepared, the sampling device continuously acquires power data which needs to be transmitted currently, and records short-term data on a local database under the condition of different time periods, power equipment and use purposes.
The test sample directly analyzes the acquired data in real time. The algorithm is also cached on a local database for subsequent verification of its performance, while also facilitating testing and analysis.
(2) Selection of number of input and output nodes and number of hidden layer neurons
The LSTM neural network is applied to the attribute analysis of the power data, and the quality of the identification and analysis result is determined by the number of the neuron nodes and the network layers. The number of input layer nodes is determined by the dimension required by the input power data characteristics, and the number of output nodes is determined by the dimension of the corresponding expected load label vector. Generally speaking, the smaller the number of neurons, the poorer the identification effect on the sample under the aliasing characteristics, and as the number of neurons increases, the identification effect will gradually increase, but if the number of neurons exceeds a certain number, the identification effect will tend to be stable. According to practical experience, 2 times or more input neurons are generally selected.
Selecting N neurons in the input layer and x time sequence variables according to different combinations of the electric power data attributes1,x2,…,xTWherein the input time sequence of the neuron at the sampling time t is
(3) Defining output layer variables
And defining prompt numbers corresponding to different electric power data attributes, performing preliminary test on the electric power data with obvious characteristics, selecting the number of neurons in an output layer, and defining output variables.
(4) Number of network layers
And for the data characteristic input signal corresponding to each data attribute, acquiring data at each sampling time point corresponding to one layer of network. The hidden layer has the function of recognizing details in the learning process of the neural network and can better distinguish. Thus, relatively speaking, the greater the number of layers, the more accurate the recognition, but the more complicated the training process. Generally, the number of hidden layers is related to the input feature type, and in the case of a single feature, one hidden layer is generally set.
(5) Determination of activation function of each layer
In an artificial neural network, an activation function represents a mapping of nodes in the network to produce respective outputs for a given input. Since the network models constructed by binary functions and linear functions have very unstable convergence characteristics, the network models are usually solved by using a canonical sigmoid activation function. The most widely used function at present is the sigmoid function. The function expression is
Where λ is 1 and b is 0.
(6) Selection of loss function
In the neural network, the difference between the ideal output value and the true value of the network model is the error, the loss function is the function describing and evaluating the difference, and the deviation degree of the model prediction output and the associated class label is quantified. In a classification model or a regression model, a supervised learning mechanism evaluates the training effect with a loss function as a standard, aiming at minimizing the difference between the predicted value and the actual value.
The selection of the loss function is in important correlation with the correction effect of the network on the weight, and the learning capability of the network is influenced to a great extent. Generally, a global continuous and differentiable loss function is selected in the optimization model, and a log-likelihood loss function (log-likelihoodloss function) is selected in the optimization model, and the expression is as follows:
the slice attributes include: slice pointer, slice length, slice volume. After the LSTM model performs data attribute analysis, the system will select a slice length and method appropriate for the data, slice the data, and edit the slice's pointers, length and volume, along with the slice contents, to synthesize a packet for network link transmission. For transmission.
According to the calculation result of the algorithm, for the length and the transmission complexity of the data packet, in the transmission mode (including but not limited to the modes of Bluetooth, WiFi, 4G/5G, local area network, mobile storage equipment and the like) with good network connection of the existing system, the type of the transmitted link is selected by combining the data receiving mode of the receiving end equipment, and especially for the link of the large-slice data selection network to transmit the slice data
And positioning and extracting a slice data packet according to the data slice method and the pointer address, and restoring slice content data through slice attributes for subsequent data storage and analysis research.
Claims (5)
1. A method for transmitting a data slice, the method comprising:
the data from the power system is acquired, and the data can be acquired through a Bluetooth mode, a WiFi mode, a 4G/5G mode, a local area network mode, a mobile storage device mode and the like, and the data type, the data size and the data length are preliminarily analyzed, and enough space is prepared for analyzing and processing the data.
2. The method according to claim 1, wherein the attributes of the power data are read, the power data are classified according to the size and length of the data, and a pre-slice model of the data slice mode is determined.
3. The method of claim 2, wherein slice properties are determined. The LSTM model is used for analyzing the attributes of the data, a slice optimization mode is selected according to the analysis result and the data length, the slice length and the number of slices are adjusted, and the purposes of dividing the original data, packaging the slices and the like in a balanced and rapid mode are achieved as far as possible.
The LSTM model is applied to solving the problems of identification and analysis of the electric power data attributes, data attribute characteristic quantities of the same type are memorized, the internal relation of input mapping to output is established, and reliable identification of the data attributes is achieved.
Compared with a shallow layer identification algorithm, the LSTM algorithm solves the problem of low identification accuracy rate caused by various attribute types; compared with the RNN model, the method can effectively solve the phenomenon of gradient disappearance in the large sample data training process in the past, can improve the reliability of attribute analysis, and provides basic support for subsequent data slicing and transmission.
The slice attributes include: slice pointer, slice length, slice volume. After the LSTM model performs data attribute analysis, the system will select a slice length and method appropriate for the data, slice the data, and edit the slice's pointers, length and volume, along with the slice contents, to synthesize a packet for network link transmission. For transmission.
4. The method according to claim 3, wherein the link type of the transmission is selected according to the algorithm calculation result, for the length and transmission complexity of the data packet, in a transmission mode (including but not limited to bluetooth, WiFi, 4G/5G, local area network, mobile storage device, etc.) with good network connection of the existing system, in combination with a data receiving mode of the receiving end device, especially for large-slice data, selecting a network link to transmit slice data.
5. The method of claim 3, wherein slice packets are located and extracted according to the data slice method and pointer address, and slice content data is restored by slice attributes for subsequent data storage and analysis research.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111270679.2A CN114143833A (en) | 2021-10-29 | 2021-10-29 | Data slice transmission method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111270679.2A CN114143833A (en) | 2021-10-29 | 2021-10-29 | Data slice transmission method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114143833A true CN114143833A (en) | 2022-03-04 |
Family
ID=80394924
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111270679.2A Pending CN114143833A (en) | 2021-10-29 | 2021-10-29 | Data slice transmission method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114143833A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110381391A (en) * | 2019-07-11 | 2019-10-25 | 北京字节跳动网络技术有限公司 | Video rapid section method, apparatus and electronic equipment |
| CN112508172A (en) * | 2020-11-23 | 2021-03-16 | 北京邮电大学 | Space flight measurement and control adaptive modulation method based on Q learning and SRNN model |
| CN112532449A (en) * | 2020-11-29 | 2021-03-19 | 国网辽宁省电力有限公司电力科学研究院 | Method for realizing selection and credible transmission of power communication slice based on 5G network |
| CN112838946A (en) * | 2020-12-17 | 2021-05-25 | 国网江苏省电力有限公司信息通信分公司 | Method for constructing intelligent sensing and early warning model based on communication network faults |
| CN113541986A (en) * | 2020-04-15 | 2021-10-22 | 中国移动通信集团浙江有限公司 | Fault prediction method and device for 5G slice and computing equipment |
-
2021
- 2021-10-29 CN CN202111270679.2A patent/CN114143833A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110381391A (en) * | 2019-07-11 | 2019-10-25 | 北京字节跳动网络技术有限公司 | Video rapid section method, apparatus and electronic equipment |
| CN113541986A (en) * | 2020-04-15 | 2021-10-22 | 中国移动通信集团浙江有限公司 | Fault prediction method and device for 5G slice and computing equipment |
| CN112508172A (en) * | 2020-11-23 | 2021-03-16 | 北京邮电大学 | Space flight measurement and control adaptive modulation method based on Q learning and SRNN model |
| CN112532449A (en) * | 2020-11-29 | 2021-03-19 | 国网辽宁省电力有限公司电力科学研究院 | Method for realizing selection and credible transmission of power communication slice based on 5G network |
| CN112838946A (en) * | 2020-12-17 | 2021-05-25 | 国网江苏省电力有限公司信息通信分公司 | Method for constructing intelligent sensing and early warning model based on communication network faults |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112084974B (en) | Multi-label rolling bearing fault diagnosis method based on meta-learning | |
| CN109543829A (en) | Method and system for hybrid deployment of deep learning neural network on terminal and cloud | |
| CN115080290B (en) | Abnormal data detection method and system based on intelligent algorithm | |
| CN109446812A (en) | A kind of embedded system firmware safety analytical method and system | |
| CN111542841A (en) | System and method for content identification | |
| CN114282579A (en) | Aviation bearing fault diagnosis method based on variational modal decomposition and residual error network | |
| US11416717B2 (en) | Classification model building apparatus and classification model building method thereof | |
| CN111078847A (en) | Power consumer intention identification method and device, computer equipment and storage medium | |
| CN111723666B (en) | Signal identification method and device based on semi-supervised learning | |
| CN110161480B (en) | Radar target identification method based on semi-supervised depth probability model | |
| CN113341919B (en) | Computing system fault prediction method based on time sequence data length optimization | |
| CN115310674A (en) | Long-time sequence prediction method based on parallel neural network model LDformer | |
| CN116416884B (en) | Testing device and testing method for display module | |
| CN107908807B (en) | Small subsample reliability evaluation method based on Bayesian theory | |
| CN115359799A (en) | Speech recognition method, training method, device, electronic equipment and storage medium | |
| CN114707041A (en) | Message recommendation method and device, computer readable medium and electronic device | |
| CN113177644A (en) | Automatic modeling system based on word embedding and depth time sequence model | |
| CN111105041B (en) | Machine learning method and device for intelligent data collision | |
| CN114143833A (en) | Data slice transmission method | |
| CN117052970A (en) | Intelligent control system and method for pneumatic ball valve assembly | |
| CN115184054B (en) | Mechanical equipment semi-supervised fault detection and analysis method, device, terminal and medium | |
| CN112685374A (en) | Log classification method and device and electronic equipment | |
| CN116183229A (en) | Vibration data feature extraction method based on self-attention mechanism of sliding window | |
| CN115630652A (en) | Customer service session emotion analysis system, method and computer system | |
| CN111798237B (en) | Abnormal transaction diagnosis method and system based on application log |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |