CN116647507A - Internet of things data processing method and system based on load self-adaption - Google Patents
Internet of things data processing method and system based on load self-adaption Download PDFInfo
- Publication number
- CN116647507A CN116647507A CN202310619803.4A CN202310619803A CN116647507A CN 116647507 A CN116647507 A CN 116647507A CN 202310619803 A CN202310619803 A CN 202310619803A CN 116647507 A CN116647507 A CN 116647507A
- Authority
- CN
- China
- Prior art keywords
- equipment
- data
- internet
- module
- things
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 76
- 238000012545 processing Methods 0.000 claims abstract description 71
- 230000000903 blocking effect Effects 0.000 claims abstract description 29
- 230000000737 periodic effect Effects 0.000 claims abstract description 10
- 238000012546 transfer Methods 0.000 claims abstract description 8
- 238000005192 partition Methods 0.000 claims description 97
- 238000004891 communication Methods 0.000 claims description 76
- 238000007781 pre-processing Methods 0.000 claims description 30
- 238000007405 data analysis Methods 0.000 claims description 27
- 238000004364 calculation method Methods 0.000 claims description 18
- 238000005516 engineering process Methods 0.000 claims description 18
- 238000010801 machine learning Methods 0.000 claims description 18
- 238000000638 solvent extraction Methods 0.000 claims description 12
- 238000010586 diagram Methods 0.000 claims description 11
- 230000006978 adaptation Effects 0.000 claims description 10
- 238000012549 training Methods 0.000 claims description 9
- 238000013138 pruning Methods 0.000 claims description 8
- 244000141353 Prunus domestica Species 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 6
- 230000006855 networking Effects 0.000 claims 1
- 230000005540 biological transmission Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000013450 outlier detection Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001550 time effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/12—Avoiding congestion; Recovering from congestion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Abstract
The invention provides a load self-adaption-based data processing method and system of the Internet of things, comprising the following steps: step S1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification; step S2: processing the equipment data according to the data blocking degree of the message queue; step S3: the device data is analyzed in an automated manner. The invention provides a method for containing a producer load self-adaptive strategy and a consumer load self-adaptive strategy of an internet of things message transfer system, which respectively aims at periodic data flow and sudden data flow of internet of things equipment and solves the technical problem of internet of things data blocking.
Description
Technical Field
The invention relates to the technical field of the Internet of things, in particular to an Internet of things data processing method and system based on load self-adaption.
Background
With the progress in broadband access networks, cloud/edge computing, big data analysis, machine learning, etc., the scale and capabilities of the internet of things are all growing dramatically. Intelligent home, intelligent health, intelligent traffic, intelligent cities, intelligent agriculture and intelligent power grids based on the internet of things are also developing vigorously. In the internet of things, a large number of heterogeneous devices constantly exchange a large amount of traffic from a network to a server. This presents both opportunities and challenges. Because big data generated by the Internet of things enable the traditional data processing capacity of the Internet of things to be gradually invalid, the development of the Internet of things needs to be promoted by combining big data technology. The internet of things data processing system faces two problems. On the one hand, the internet of things data processing system needs to provide a powerful analysis function and comprehensively analyze equipment data of multiple sources, however, the current internet of things data processing system is single in application scene or needs complicated manual operation, and the efficiency is unsatisfactory. On the other hand, the data processing system of the internet of things needs to have the processing capability of coping with a large amount of equipment data streams of the internet of things, however, the characteristics of the data of the internet of things are that equipment loads are different and change, different equipment data flows are different, the same equipment data flow also changes suddenly, the problem of data blocking can be caused, and the efficiency of subsequent data analysis and processing is affected.
Aiming at the problem of data analysis of the Internet of things, the Internet of things is usually subjected to predictive analysis by using a machine learning technical means. Machine learning utilizes information hidden in big data and extracts value from big data sources with minimal effort. The machine learning is very suitable for the environment of the Internet of things, because the Internet of things has different data sources, the data volume is huge, the variety is great, and the required comprehensive analysis is manpower-insufficient. However, the existing internet of things machine learning technology is insufficient to meet the requirement of internet of things data analysis.
Aiming at the problem of data blocking of the Internet of things, two types of technical means are generally adopted, one type is to reasonably allocate available resources of different data streams through a load balancing strategy, and the other type is to reduce the flow of high-speed data streams through resampling or data reduction technology. The network resource load balancing strategy aims at improving the transmission efficiency of data flow transmitted from equipment to the system port of the Internet of things through the network, and achieves good performance in solving the problem of data blocking. Adaptive sampling and data reduction techniques reduce the amount of data in the device stream during the different stages of data stream generation and transmission, both of which achieve good performance over most data streams. However, the existing internet of things data blocking technology cannot completely solve the problem of internet of things data blocking.
The existing methods of the internet of things machine learning technology and the internet of things data blocking technology at least have the following technical problems:
1. for the problem of data analysis of the Internet of things, the existing machine learning technology of the Internet of things requires participants to master a great deal of expertise, compare the performances of different models, consider different algorithms, perform complicated hyper-parameter adjustment and take a long training time.
2. For the problem of data blocking of the internet of things, the existing network resource load balancing strategy focuses more on the blocking condition of data on the network, the data flow arrives at the internet of things system by the equipment and is only part of data transmission of the internet of things, and the data blocking can also occur in the internet of things data processing system. In the scenario that network transmission is not a bottleneck, the effectiveness of the network load balancing policy is not significant, and the data blocking problem inside the system is more of concern for the data processing system of the internet of things.
3. For the problem of data blocking of the Internet of things, but for data streams with large data variability, each mutation value is difficult to accurately capture. Furthermore, the reduced data volume also affects the accuracy of the subsequent data analysis phase, which is more pronounced in the presence of highly variable data streams.
Therefore, the method in the prior art has the problems of the machine learning technology of the Internet of things and the data blocking technology of the Internet of things.
Patent document CN115865982a discloses an internet of things system and an internet of things data processing method, wherein the internet of things system comprises: the user terminal is configured to acquire a private key from the internet server according to an account of the user, and generate a data packet by utilizing the private key based on uploading information of the user, wherein the data packet comprises the uploading information and a signature generated according to the uploading information; the client is further configured to send the data packet to a blockchain platform; the Internet of things server is used for distributing asymmetric encryption keys. But the invention does not solve the problem of data blocking of the internet of things.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a load self-adaption-based data processing method and system for the Internet of things.
The invention provides a load self-adaption-based data processing method of the Internet of things, which comprises the following steps:
step S1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification;
Step S2: processing the equipment data according to the data blocking degree of the message queue;
step S3: the device data is analyzed in an automated manner.
Preferably, in said step S1:
the message producer load self-adaptive strategy obtains equipment period information of the Internet of things equipment, and classifies the Internet of things equipment according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the classification of the Internet of things equipment;
the message producer load self-adaption strategy method specifically comprises the following steps:
step S1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all equipment communication frequency sets of the Internet of things to the background;
step S1.2: detecting outliers in the equipment communication frequency set, classifying the equipment and storing the equipment classification;
a device for finding out from a device frequency dataset that the periodic data flow is above a preset criterion, comprising:
step s1.2.1: calculating an outlier threshold of device frequencies in the device communication frequency set;
step S1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than an outlier threshold;
step S1.2.3: if the device communication frequency is greater than the outlier threshold, marking the device as an outlier device; if the communication frequency of the equipment is smaller than or equal to the outlier threshold, the equipment is marked as common equipment;
As an outlier threshold calculation method, stepStep S1.2.1, calculating outlier threshold F by using a box line graph method 3 The formula is:
F 3 ＝Q 3 +1.5IQR
wherein Q is 3 The third quartile is the device communication frequency set, and the IQR is the quartile range of the device communication frequency set;
step S1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold value as the maximum partition key of the message queue;
step S1.4: the message producer of the message queue randomly selects one partition key from 1 to the largest partition key and sends the device data to the partition of the message queue.
Preferably, in said step S2:
optimizing a message transfer system of the data processing system of the Internet of things, and distributing consumers according to the partition hysteresis degree of a message queue:
step S2.1: calculating the partition message hysteresis degree of the message queue;
the method for calculating the hysteresis degree of the partition message comprises the following steps:
step S2.1.1: calculating a partition current message lag value lag t ：
lag t ＝offset end -offset current
Wherein, the offset end Offset for partitioning the current message end offset current The offset has been consumed for the current message for the partition;
step S2.1.2: recalculating partition current message hysteresis based on current hysteresis value, calculating partition message hysteresis EWMA using exponentially weighted moving average t ：
EWMA t ＝α×lag t +(1-α)×EWMA t-1
Wherein, lag t For partitioning the current message hysteresis value, α is the weight of the current message hysteresis value, the value is between 0 and 1, EWMA t-1 A message hysteresis level for the last calculation;
step S2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
step S2.3: if the hysteresis level exceeds the threshold, the message hysteresis level is ranked from high to low, and the consumers are ranked from low to high by the number of consuming partitions, and the consumers of the message queue partitions are reassigned.
Preferably, in said step S3:
the automatic data analysis module analyzes the equipment data in an automatic mode:
step S3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
step S3.1.1: processing the device history data set using a sliding window;
step S3.1.2: pruning noise data of the device history dataset;
step S3.1.3: normalizing the processing device history dataset;
step S3.1.4: feature selection is performed on the device history dataset.
Step S3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing the model in a background;
step S3.3: predicting the current data of the equipment by using the model;
Step S3.3.1: preprocessing current data of the equipment by using a method for preprocessing a historical data set of the equipment;
step S3.3.2: the current data of the device is predicted by using the model stored in the background.
Preferably, the method of sliding window processing device history data set in step S3.1.1 comprises:
step S3.1.1.1: moving the sliding window to the first untreated row of the device history data set, and averaging all rows within the window size range;
step S3.1.1.2: repeating step S3.1.1.1 until the device history data sets are all processed;
step S3.1.2 prunes noise data in the device history data set, comprising:
step S3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
step S3.1.2.2: deleting a fixed proportion of rows from the last row of the historical data set of the device;
the standardized processing equipment history data z in step S3.1.3 is calculated by:
wherein x is data in the equipment history data set, mu is the mean value of the equipment history data set, and s is the standard deviation of the equipment history data set;
the method of feature selection of the device history dataset in step S3.1.4 includes:
step S3.1.4.1: calculating mutual information of each feature of the historical data set of the equipment and the predicted target feature;
Step S3.1.4.2: sorting the characteristics of the historical data set of the equipment according to the mutual information;
step S3.1.4.3: the feature with the lowest mutual information of fixed proportion is deleted.
According to the invention, the data processing system of the Internet of things based on load self-adaption comprises:
module M1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification;
module M2: processing the equipment data according to the data blocking degree of the message queue;
module M3: the device data is analyzed in an automated manner.
Preferably, in said module M1:
the message producer load self-adaptive strategy obtains equipment period information of the Internet of things equipment, and classifies the Internet of things equipment according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the classification of the Internet of things equipment;
the message producer load self-adaption strategy method specifically comprises the following steps:
module M1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all equipment communication frequency sets of the Internet of things to the background;
Module M1.2: detecting outliers in the equipment communication frequency set, classifying the equipment and storing the equipment classification;
a device for finding out from a device frequency dataset that the periodic data flow is above a preset criterion, comprising:
module M1.2.1: calculating an outlier threshold of device frequencies in the device communication frequency set;
module M1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than an outlier threshold;
module M1.2.3: if the device communication frequency is greater than the outlier threshold, marking the device as an outlier device; if the communication frequency of the equipment is smaller than or equal to the outlier threshold, the equipment is marked as common equipment;
as an outlier threshold calculation method, the module M1.2.1 calculates an outlier threshold F using a box-line diagram method 3 The formula is:
F 3 ＝Q 3 +1.5IQR
wherein Q is 3 The third quartile is the device communication frequency set, and the IQR is the quartile range of the device communication frequency set;
module M1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold value as the maximum partition key of the message queue;
module M1.4: the message producer of the message queue randomly selects one partition key from 1 to the largest partition key and sends the device data to the partition of the message queue.
Preferably, in said module M2:
Optimizing a message transfer system of the data processing system of the Internet of things, and distributing consumers according to the partition hysteresis degree of a message queue:
module M2.1: calculating the partition message hysteresis degree of the message queue;
the method for calculating the hysteresis degree of the partition message comprises the following steps:
module M2.1.1: computing partition current message hysteresisValue lag t ：
lag t ＝offset end -offset current
Wherein, the offset end Offset for partitioning the current message end offset current The offset has been consumed for the current message for the partition;
module M2.1.2: recalculating partition current message hysteresis based on current hysteresis value, calculating partition message hysteresis EWMA using exponentially weighted moving average t ：
EWMA t ＝α×lag t +(1-α)×EWMA t-1
Wherein, lag t For partitioning the current message hysteresis value, α is the weight of the current message hysteresis value, the value is between 0 and 1, EWMA t-1 A message hysteresis level for the last calculation;
module M2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
module M2.3: if the hysteresis level exceeds the threshold, the message hysteresis level is ranked from high to low, and the consumers are ranked from low to high by the number of consuming partitions, and the consumers of the message queue partitions are reassigned.
Preferably, in said module M3:
the automatic data analysis module analyzes the equipment data in an automatic mode:
Module M3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
module M3.1.1: processing the device history data set using a sliding window;
module M3.1.2: pruning noise data of the device history dataset;
module M3.1.3: normalizing the processing device history dataset;
module M3.1.4: feature selection is performed on the device history dataset.
Module M3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing the model in a background;
module M3.3: predicting the current data of the equipment by using the model;
module M3.3.1: preprocessing current data of the equipment by using a method for preprocessing a historical data set of the equipment;
module M3.3.2: the current data of the device is predicted by using the model stored in the background.
Preferably, the method of sliding window processing device history data set in module M3.1.1 comprises:
module M3.1.1.1: moving the sliding window to the first untreated row of the device history data set, and averaging all rows within the window size range;
module M3.1.1.2: repeating module M3.1.1.1 until the device history data sets are all processed;
the module M3.1.2 prunes noise data in the device history data set, comprising:
Module M3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
module M3.1.2.2: deleting a fixed proportion of rows from the last row of the historical data set of the device;
the standardized processing equipment history data z in the module M3.1.3 is calculated by the following steps:
wherein x is data in the equipment history data set, mu is the mean value of the equipment history data set, and s is the standard deviation of the equipment history data set;
the method of feature selection of the device history dataset in module M3.1.4 includes:
module M3.1.4.1: calculating mutual information of each feature of the historical data set of the equipment and the predicted target feature;
module M3.1.4.2: sorting the characteristics of the historical data set of the equipment according to the mutual information;
module M3.1.4.3: the feature with the lowest mutual information of fixed proportion is deleted.
Compared with the prior art, the invention has the following beneficial effects:
1. the data analysis and prediction result has high accuracy, is simple and easy to use, has quick data processing response and high throughput, and is not easily influenced by data blockage in the system;
2. the invention provides an automatic data analysis method of the Internet of things, which comprises data preprocessing, automatic model training and predictive analysis, provides accuracy for users, ensures higher convenience and solves the problem of data analysis of the Internet of things;
3. The invention provides a method for containing a producer load self-adaptive strategy and a consumer load self-adaptive strategy of an internet of things message transfer system, which respectively aims at periodic data flow and sudden data flow of internet of things equipment and solves the technical problem of internet of things data blocking.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a specific flowchart of an Internet of things data processing system based on load adaptation provided by the invention;
FIG. 2 is a schematic workflow diagram of a load adaptation strategy for a producer of an Internet of things messaging system in accordance with an embodiment of the present invention;
FIG. 3 is a schematic workflow diagram of a consumer load adaptation strategy for an Internet of things messaging system in accordance with an embodiment of the present invention;
fig. 4 is a schematic workflow diagram of automated data analysis of the internet of things in an embodiment of the invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
Example 1:
the invention discloses a load self-adaption-based data processing system and method of the Internet of things, wherein in the aspect of data analysis, the method combines the technology of the Internet of things and automatic machine learning, and a data preprocessing module is added to make up for the defect of automatic machine learning, so that accurate and convenient data prediction analysis capability is provided for users; secondly, for the problem of data blocking in the Internet of things, the method adds two different load self-adaptive strategies according to the characteristics of periodicity and burstiness of the data flow of the Internet of things equipment.
The invention provides an Internet of things data processing system and method based on load self-adaption. Firstly, an automatic data analysis method of the Internet of things provides an automatic data analysis service with universality, convenience and high performance for users of the Internet of things with data analysis requirements; and then, through two load self-adaptive strategies of the Internet of things, the problem of data blocking caused by the difference and change of equipment loads in the Internet of things is solved. The automatic data analysis method can provide a convenient and quick choice with higher accuracy for the user in big data analysis. The producer load self-adaptive strategy and the consumer load self-adaptive strategy enable the response speed of the data transmission system of the Internet of things to be faster and the throughput to be higher.
According to the data processing method of the internet of things based on load self-adaption, as shown in fig. 1-4, the data processing method comprises the following steps:
step S1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification;
specifically, in the step S1:
the message producer load self-adaptive strategy obtains equipment period information of the Internet of things equipment, and classifies the Internet of things equipment according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the classification of the Internet of things equipment;
the message producer load self-adaption strategy method specifically comprises the following steps:
step S1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all equipment communication frequency sets of the Internet of things to the background;
step S1.2: detecting outliers in the equipment communication frequency set, classifying the equipment and storing the equipment classification;
a device for finding out from a device frequency dataset that the periodic data flow is above a preset criterion, comprising:
step s1.2.1: calculating an outlier threshold of device frequencies in the device communication frequency set;
Step S1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than an outlier threshold;
step S1.2.3: if the device communication frequency is greater than the outlier threshold, marking the device as an outlier device; if the communication frequency of the equipment is smaller than or equal to the outlier threshold, the equipment is marked as common equipment;
as an outlier threshold calculation method, step s1.2.1 calculates an outlier threshold F using a box-line diagram method 3 The formula is:
F 3 ＝Q 3 +1.5IQR
wherein Q is 3 The third quartile is the device communication frequency set, and the IQR is the quartile range of the device communication frequency set;
step S1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold value as the maximum partition key of the message queue;
step S1.4: the message producer of the message queue randomly selects one partition key from 1 to the largest partition key and sends the device data to the partition of the message queue.
Step S2: processing the equipment data according to the data blocking degree of the message queue;
specifically, in the step S2:
optimizing a message transfer system of the data processing system of the Internet of things, and distributing consumers according to the partition hysteresis degree of a message queue:
step S2.1: calculating the partition message hysteresis degree of the message queue;
the method for calculating the hysteresis degree of the partition message comprises the following steps:
Step S2.1.1: calculating a partition current message lag value lag t ：
lag t ＝offset end -offset current
Wherein, the offset end Offset for partitioning the current message end offset current The offset has been consumed for the current message for the partition;
step S2.1.2: recalculating partition current message hysteresis based on current hysteresis value, calculating partition message hysteresis EWMA using exponentially weighted moving average t ：
EWMA t ＝α×lag t +(1-α)×EWMA t-1
Wherein, lag t For partitioning the current message hysteresis value, α is the weight of the current message hysteresis value, the value is between 0 and 1, EWMA t-1 A message hysteresis level for the last calculation;
step S2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
step S2.3: if the hysteresis level exceeds the threshold, the message hysteresis level is ranked from high to low, and the consumers are ranked from low to high by the number of consuming partitions, and the consumers of the message queue partitions are reassigned.
Step S3: the device data is analyzed in an automated manner.
Specifically, in the step S3:
the automatic data analysis module analyzes the equipment data in an automatic mode:
step S3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
step S3.1.1: processing the device history data set using a sliding window;
Step S3.1.2: pruning noise data of the device history dataset;
step S3.1.3: normalizing the processing device history dataset;
step S3.1.4: feature selection is performed on the device history dataset.
Step S3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing the model in a background;
step S3.3: predicting the current data of the equipment by using the model;
step S3.3.1: preprocessing current data of the equipment by using a method for preprocessing a historical data set of the equipment;
step S3.3.2: the current data of the device is predicted by using the model stored in the background.
Specifically, the method of sliding window processing device history data set in step S3.1.1 includes:
step S3.1.1.1: moving the sliding window to the first untreated row of the device history data set, and averaging all rows within the window size range;
step S3.1.1.2: repeating step S3.1.1.1 until the device history data sets are all processed;
step S3.1.2 prunes noise data in the device history data set, comprising:
step S3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
step S3.1.2.2: deleting a fixed proportion of rows from the last row of the historical data set of the device;
The standardized processing equipment history data z in step S3.1.3 is calculated by:
wherein x is data in the equipment history data set, mu is the mean value of the equipment history data set, and s is the standard deviation of the equipment history data set;
the method of feature selection of the device history dataset in step S3.1.4 includes:
step S3.1.4.1: calculating mutual information of each feature of the historical data set of the equipment and the predicted target feature;
step S3.1.4.2: sorting the characteristics of the historical data set of the equipment according to the mutual information;
step S3.1.4.3: the feature with the lowest mutual information of fixed proportion is deleted.
Example 2:
example 2 is a preferable example of example 1 to more specifically explain the present invention.
The invention also provides a load-adaptive-based internet of things data processing system, which can be realized by executing the flow steps of the load-adaptive-based internet of things data processing method, namely, a person skilled in the art can understand the load-adaptive-based internet of things data processing method as a preferred implementation mode of the load-adaptive-based internet of things data processing system.
According to the invention, the data processing system of the Internet of things based on load self-adaption comprises:
module M1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification;
specifically, in the module M1:
the message producer load self-adaptive strategy obtains equipment period information of the Internet of things equipment, and classifies the Internet of things equipment according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the classification of the Internet of things equipment;
the message producer load self-adaption strategy method specifically comprises the following steps:
module M1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all equipment communication frequency sets of the Internet of things to the background;
module M1.2: detecting outliers in the equipment communication frequency set, classifying the equipment and storing the equipment classification;
a device for finding out from a device frequency dataset that the periodic data flow is above a preset criterion, comprising:
module M1.2.1: calculating an outlier threshold of device frequencies in the device communication frequency set;
Module M1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than an outlier threshold;
module M1.2.3: if the device communication frequency is greater than the outlier threshold, marking the device as an outlier device; if the communication frequency of the equipment is smaller than or equal to the outlier threshold, the equipment is marked as common equipment;
as an outlier threshold calculation method, the module M1.2.1 calculates an outlier threshold F using a box-line diagram method 3 The formula is:
F 3 ＝Q 3 +1.5IQR
wherein Q is 3 The third quartile is the device communication frequency set, and the IQR is the quartile range of the device communication frequency set;
module M1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold value as the maximum partition key of the message queue;
module M1.4: the message producer of the message queue randomly selects one partition key from 1 to the largest partition key and sends the device data to the partition of the message queue.
Module M2: processing the equipment data according to the data blocking degree of the message queue;
specifically, in the module M2:
optimizing a message transfer system of the data processing system of the Internet of things, and distributing consumers according to the partition hysteresis degree of a message queue:
module M2.1: calculating the partition message hysteresis degree of the message queue;
the method for calculating the hysteresis degree of the partition message comprises the following steps:
Module M2.1.1: calculating a partition current message lag value lag t ：
lag t ＝offset end -offset current
Wherein, the offset end Offset for partitioning the current message end offset current The offset has been consumed for the current message for the partition;
module M2.1.2: heavy according to current hysteresis valueNew calculation of partition current message hysteresis level, calculation of partition message hysteresis level EWMA using exponentially weighted moving average t ：
EWMA t ＝α×lag t +(1-α)×EWMA t-1
Wherein, lag t For partitioning the current message hysteresis value, θ is the weight of the current message hysteresis value, with values between 0 and 1, EWMA t-1 A message hysteresis level for the last calculation;
module M2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
module M2.3: if the hysteresis level exceeds the threshold, the message hysteresis level is ranked from high to low, and the consumers are ranked from low to high by the number of consuming partitions, and the consumers of the message queue partitions are reassigned.
Module M3: the device data is analyzed in an automated manner.
Specifically, in the module M3:
the automatic data analysis module analyzes the equipment data in an automatic mode:
module M3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
module M3.1.1: processing the device history data set using a sliding window;
Module M3.1.2: pruning noise data of the device history dataset;
module M3.1.3: normalizing the processing device history dataset;
module M3.1.4: feature selection is performed on the device history dataset.
Module M3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing the model in a background;
module M3.3: predicting the current data of the equipment by using the model;
module M3.3.1: preprocessing current data of the equipment by using a method for preprocessing a historical data set of the equipment;
module M3.3.2: the current data of the device is predicted by using the model stored in the background.
Specifically, the method of sliding window processing device history data set in module M3.1.1 includes:
module M3.1.1.1: moving the sliding window to the first untreated row of the device history data set, and averaging all rows within the window size range;
module M3.1.1.2: repeating module M3.1.1.1 until the device history data sets are all processed;
the module M3.1.2 prunes noise data in the device history data set, comprising:
module M3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
module M3.1.2.2: deleting a fixed proportion of rows from the last row of the historical data set of the device;
The standardized processing equipment history data z in the module M3.1.3 is calculated by the following steps:
wherein x is data in the equipment history data set, mu is the mean value of the equipment history data set, and s is the standard deviation of the equipment history data set;
the method of feature selection of the device history dataset in module M3.1.4 includes:
module M3.1.4.1: calculating mutual information of each feature of the historical data set of the equipment and the predicted target feature;
module M3.1.4.2: sorting the characteristics of the historical data set of the equipment according to the mutual information;
module M3.1.4.3: the feature with the lowest mutual information of fixed proportion is deleted.
Example 3:
example 3 is a preferable example of example 1 to more specifically explain the present invention.
The invention provides an Internet of things data processing system and method based on load self-adaption, which are used for solving or at least partially solving the technical problems of Internet of things data analysis and Internet of things data blocking. Through integrating automatic machine learning technology, the method improves various data preprocessing schemes, realizes universal service suitable for data analysis in various fields of the Internet of things, and provides higher convenience for users of the Internet of things. By optimizing a message passing system in the data processing system of the Internet of things, two load self-adaptive strategies of a producer and a consumer are provided. These load adaptation strategies can collectively improve the overall performance of the system.
The invention provides an internet of things data processing system and method based on load self-adaption, and referring to fig. 1, a specific flow chart of the internet of things data processing system and method based on load self-adaption is provided. The scheme comprises the following steps:
firstly, an Internet of things data processing system comprises a plurality of data processing modules; the basic steps are as follows:
s1: the method comprises the steps that a message producer load self-adaptive strategy obtains equipment period information of Internet of things equipment, and the Internet of things equipment is classified according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and issuing the equipment data to a message queue according to the classification of the equipment;
s2: the message consumer load self-adaptive strategy processes the equipment data according to the data blocking degree of the message queue;
s3: an automated data analysis module analyzes the device data in an automated manner.
As a producer load self-adaptive strategy, S1 optimizes a message transmission system of a data processing system of the Internet of things, and distributes available partition numbers for equipment according to equipment communication periods, so as to further relieve the problem of data blocking caused by periodic flow:
the S1 message producer load self-adaption strategy method specifically comprises the following steps:
S1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all the equipment communication frequency sets of the Internet of things to the background;
s1.2: detecting outliers in the device communication frequency set, classifying the devices, and storing the device classifications;
s1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold as the maximum partition key of the message queue;
s1.4: the message producer of the message queue randomly selects one partition key from 1 to the maximum partition key and sends the device data to the partition of the message queue.
As an outlier detection method, S1.2 finds out a device with too high periodic data traffic from the device frequency dataset, comprising:
s1.2.1: calculating an outlier threshold of the device frequency in the device communication frequency set;
s1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than the outlier threshold;
s1.2.3: if yes, the equipment is marked as outlier equipment; if not, marking the equipment as common equipment; as an outlier threshold calculation method, s1.2.1 calculates an outlier threshold by using a box line graph method, and the formula is:
F 3 ＝Q 3 +1.5IQR
Wherein Q is 3 For the third quartile of the set of device communication frequencies, IQR is the quartile range of the set of device communication frequencies.
As a consumer load self-adaptive strategy, S2 optimizes a message transmission system of the data processing system of the Internet of things, distributes consumers according to the partition hysteresis degree of a message queue, and further relieves the problem of data blocking caused by sudden traffic:
specifically, referring to fig. 2, a workflow diagram of an internet of things messaging system producer load adaptation strategy is shown.
S2, the method for processing the equipment data according to the data blocking degree of the message queue specifically comprises the following steps:
s2.1: calculating the partition message hysteresis degree of the message queue;
s2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
s2.3: and if so, ordering the message hysteresis degree from high to low, and reallocating the consumers of the message queue partitions according to the ordering of the number of the consumption partitions from low to high.
As a method for calculating the hysteresis level of the partition message, the method for calculating the hysteresis level of the partition message in S2.1 includes:
s2.1.1: calculating the current message hysteresis value of the partition;
S2.1.2: and recalculating the current message hysteresis degree of the partition according to the current hysteresis value.
As a calculation method of the partition message hysteresis value, the formula of S2.1.1 is:
lag t ＝offset end -offset current
wherein offset is end Ending the offset for the partition current message current An offset has been consumed for the partition's current message.
As a calculation method of the partition message hysteresis level considering the time effect, S2.1.2 calculates the partition message hysteresis level using an exponentially weighted moving average method, the formula is:
EWMA t ＝α×lag t +(1-α)×EWMA t-1
wherein lag t For the partition current message hysteresis value, α is the weight of the current message hysteresis value, the value is between 0 and 1, EWMA t-1 The degree of message hysteresis for the last calculation.
Specifically, please refer to fig. 3, which is a schematic diagram of a workflow of the internet of things messaging system consumer load adaptation policy.
As an automated data analysis technique, the method for analyzing the device data by the automated manner S3 specifically includes:
s3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
s3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing a model in a background;
S3.3: predicting current data of the device using the model.
As a method of preprocessing a device history data set, the method of preprocessing a device history data set in S3.1 includes:
s3.1.1: processing the device history data set using a sliding window;
s3.1.2: pruning noise data of the device history dataset;
s3.1.3: normalizing the device history dataset;
s3.1.4: and selecting the characteristics of the historical data set of the equipment.
As a method of sliding window preprocessing a device history data set, the method of sliding window processing the device history data set in S3.1.1 includes:
s3.1.1.1: the sliding window is moved to the first line of the device history data set that has not been processed and all lines within the window size range are averaged.
S3.1.1.2: s3.1.1.1 is repeated until the device history data sets are all processed.
As a method of preprocessing a device history data set by pruning, S3.1.2 prunes noise data in the device history data set, comprising:
s3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
s3.1.2.2: and deleting a fixed proportion of rows from the last row of the device history data set.
As a method for normalizing the preprocessing device history data set, the normalization processing device history data in S3.1.3 is calculated in the following manner:
where x is the data in the device history data set, μ is the mean of the device history data set, and s is the standard deviation of the device history data set.
As a method of feature selection preprocessing a device history data set, the method of feature selection of the device history data set in S3.1.4 includes:
s3.1.4.1: calculating mutual information of each feature of the equipment historical data set and the predicted target feature;
s3.1.4.2: sorting the characteristics of the device history dataset according to the mutual information;
s3.1.4.3: and deleting the characteristic with the lowest mutual information in a fixed proportion.
As a method for predicting a data set in real time, the method for predicting current data of the device in S3.3 includes:
s3.3.1: preprocessing the current data of the equipment by using the method for preprocessing the historical data set of the equipment;
s3.3.2: predicting the current data of the device using the model stored in the background.
Specifically, please refer to fig. 4, which is a schematic diagram of a workflow of automated data analysis of the internet of things.
Those skilled in the art will appreciate that the systems, apparatus, and their respective modules provided herein may be implemented entirely by logic programming of method steps such that the systems, apparatus, and their respective modules are implemented as logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc., in addition to the systems, apparatus, and their respective modules being implemented as pure computer readable program code. Therefore, the system, the apparatus, and the respective modules thereof provided by the present application may be regarded as one hardware component, and the modules included therein for implementing various programs may also be regarded as structures within the hardware component; modules for implementing various functions may also be regarded as being either software programs for implementing the methods or structures within hardware components.
The foregoing describes specific embodiments of the present application. It is to be understood that the application is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the application. The embodiments of the application and the features of the embodiments may be combined with each other arbitrarily without conflict.
Claims (10)
1. The data processing method of the internet of things based on load self-adaption is characterized by comprising the following steps of:
step S1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification;
step S2: processing the equipment data according to the data blocking degree of the message queue;
step S3: the device data is analyzed in an automated manner.
2. The internet of things data processing method based on load adaptation according to claim 1, wherein in the step S1:
the message producer load self-adaptive strategy obtains equipment period information of the Internet of things equipment, and classifies the Internet of things equipment according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the classification of the Internet of things equipment;
the message producer load self-adaption strategy method specifically comprises the following steps:
step S1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all equipment communication frequency sets of the Internet of things to the background;
step S1.2: detecting outliers in the equipment communication frequency set, classifying the equipment and storing the equipment classification;
A device for finding out from a device frequency dataset that the periodic data flow is above a preset criterion, comprising:
step s1.2.1: calculating an outlier threshold of device frequencies in the device communication frequency set;
step S1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than an outlier threshold;
step S1.2.3: if the device communication frequency is greater than the outlier threshold, marking the device as an outlier device; if the communication frequency of the equipment is smaller than or equal to the outlier threshold, the equipment is marked as common equipment;
as an outlier threshold calculation method, step s1.2.1 calculates an outlier threshold F using a box-line diagram method 3 The formula is:
F 3 ＝ 3 +1.5IQR
wherein Q is 3 The third quartile is the device communication frequency set, and the IQR is the quartile range of the device communication frequency set;
step S1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold value as the maximum partition key of the message queue;
step S1.4: the message producer of the message queue randomly selects one partition key from 1 to the largest partition key and sends the device data to the partition of the message queue.
3. The internet of things data processing method based on load adaptation according to claim 1, wherein in the step S2:
Optimizing a message transfer system of the data processing system of the Internet of things, and distributing consumers according to the partition hysteresis degree of a message queue:
step S2.1: calculating the partition message hysteresis degree of the message queue;
the method for calculating the hysteresis degree of the partition message comprises the following steps:
step S2.1.1: calculating a partition current message lag value lag t ：
lag t ＝offset end -offset current
Wherein, the offset end Offset for partitioning the current message end offset current The offset has been consumed for the current message for the partition;
step S2.1.2: recalculating partition current message hysteresis based on current hysteresis value, calculating partition message hysteresis EWMA using exponentially weighted moving average t ：
EWMA t ＝×lag t +(1-)×EWMA t-1
Wherein, lag t For partitioning the current message hysteresis value, α is the weight of the current message hysteresis value, the value is between 0 and 1, EWMA t-1 A message hysteresis level for the last calculation;
step S2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
step S2.3: if the hysteresis level exceeds the threshold, the message hysteresis level is ranked from high to low, and the consumers are ranked from low to high by the number of consuming partitions, and the consumers of the message queue partitions are reassigned.
4. The internet of things data processing method based on load adaptation according to claim 1, wherein in the step S3:
The automatic data analysis module analyzes the equipment data in an automatic mode:
step S3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
step S3.1.1: processing the device history data set using a sliding window;
step S3.1.2: pruning noise data of the device history dataset;
step S3.1.3: normalizing the processing device history dataset;
step S3.1.4: feature selection is performed on the device history dataset.
Step S3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing the model in a background;
step S3.3: predicting the current data of the equipment by using the model;
step S3.3.1: preprocessing current data of the equipment by using a method for preprocessing a historical data set of the equipment;
step S3.3.2: the current data of the device is predicted by using the model stored in the background.
5. The internet of things data processing method based on load adaptation according to claim 4, wherein:
the method of sliding window processing device history data set in step S3.1.1 includes:
step S3.1.1.1: moving the sliding window to the first untreated row of the device history data set, and averaging all rows within the window size range;
Step S3.1.1.2: repeating step S3.1.1.1 until the device history data sets are all processed;
step S3.1.2 prunes noise data in the device history data set, comprising:
step S3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
step S3.1.2.2: deleting a fixed proportion of rows from the last row of the historical data set of the device;
the standardized processing equipment history data z in step S3.1.3 is calculated by:
wherein x is data in the equipment history data set, mu is the mean value of the equipment history data set, and s is the standard deviation of the equipment history data set;
the method of feature selection of the device history dataset in step S3.1.4 includes:
step S3.1.4.1: calculating mutual information of each feature of the historical data set of the equipment and the predicted target feature;
step S3.1.4.2: sorting the characteristics of the historical data set of the equipment according to the mutual information;
step S3.1.4.3: the feature with the lowest mutual information of fixed proportion is deleted.
6. The utility model provides a thing networking data processing system based on load self-adaptation which characterized in that includes:
module M1: acquiring equipment period information of the Internet of things equipment, classifying the Internet of things equipment, receiving equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the Internet of things equipment classification;
Module M2: processing the equipment data according to the data blocking degree of the message queue;
module M3: the device data is analyzed in an automated manner.
7. The load-adaptive-based internet of things data processing system according to claim 6, wherein in the module M1:
the message producer load self-adaptive strategy obtains equipment period information of the Internet of things equipment, and classifies the Internet of things equipment according to the equipment period information; receiving the equipment data uploaded by the Internet of things equipment, and publishing the equipment data to a message queue according to the classification of the Internet of things equipment;
the message producer load self-adaption strategy method specifically comprises the following steps:
module M1.1: calculating equipment communication frequency according to the period information of the equipment of the Internet of things, and storing all equipment communication frequency sets of the Internet of things to the background;
module M1.2: detecting outliers in the equipment communication frequency set, classifying the equipment and storing the equipment classification;
a device for finding out from a device frequency dataset that the periodic data flow is above a preset criterion, comprising:
module M1.2.1: calculating an outlier threshold of device frequencies in the device communication frequency set;
module M1.2.2: checking whether each device communication frequency of the set of device communication frequencies is greater than an outlier threshold;
Module M1.2.3: if the device communication frequency is greater than the outlier threshold, marking the device as an outlier device; if the communication frequency of the equipment is smaller than or equal to the outlier threshold, the equipment is marked as common equipment;
as an outlier threshold calculation method, the module M1.2.1 calculates an outlier threshold F using a box-line diagram method 3 The formula is:
F 3 ＝ 3 +1.5IQR
wherein Q is 3 The third quartile is the device communication frequency set, and the IQR is the quartile range of the device communication frequency set;
module M1.3: calculating the ratio of the communication frequency of the equipment to the outlier threshold value as the maximum partition key of the message queue;
module M1.4: the message producer of the message queue randomly selects one partition key from 1 to the largest partition key and sends the device data to the partition of the message queue.
8. The load-adaptive-based internet of things data processing system according to claim 6, wherein in the module M2:
optimizing a message transfer system of the data processing system of the Internet of things, and distributing consumers according to the partition hysteresis degree of a message queue:
module M2.1: calculating the partition message hysteresis degree of the message queue;
the method for calculating the hysteresis degree of the partition message comprises the following steps:
module M2.1.1: calculating a partition current message lag value lag t ：
lag t ＝offset end -offset current
Wherein, the offset end Offset for partitioning the current message end offset current The offset has been consumed for the current message for the partition;
module M2.1.2: recalculating partition current message hysteresis based on current hysteresis value, calculating partition message hysteresis EWMA using exponentially weighted moving average t ：
EWMA t ＝×lag t +(1-)×EWMA t-1
Wherein, lag t For partitioning the current message hysteresis value, α is the weight of the current message hysteresis value, the value is between 0 and 1, EWMA t-1 A message hysteresis level for the last calculation;
module M2.2: the consumer group of the message queue monitors whether the hysteresis degree of the partitioned message exceeds a threshold value;
module M2.3: if the hysteresis level exceeds the threshold, the message hysteresis level is ranked from high to low, and the consumers are ranked from low to high by the number of consuming partitions, and the consumers of the message queue partitions are reassigned.
9. The load-adaptive-based internet of things data processing system according to claim 6, wherein in the module M3:
the automatic data analysis module analyzes the equipment data in an automatic mode:
module M3.1: preprocessing the equipment history data set and storing the preprocessed equipment history data set as a preprocessed equipment data set;
module M3.1.1: processing the device history data set using a sliding window;
Module M3.1.2: pruning noise data of the device history dataset;
module M3.1.3: normalizing the processing device history dataset;
module M3.1.4: feature selection is performed on the device history dataset.
Module M3.2: model training the preprocessed device data set by using an automatic machine learning technology, and storing the model in a background;
module M3.3: predicting the current data of the equipment by using the model;
module M3.3.1: preprocessing current data of the equipment by using a method for preprocessing a historical data set of the equipment;
module M3.3.2: the current data of the device is predicted by using the model stored in the background.
10. The load-adaptive-based internet of things data processing system of claim 9, wherein:
the method of the sliding window processing device history data set in module M3.1.1 includes:
module M3.1.1.1: moving the sliding window to the first untreated row of the device history data set, and averaging all rows within the window size range;
module M3.1.1.2: repeating module M3.1.1.1 until the device history data sets are all processed;
the module M3.1.2 prunes noise data in the device history data set, comprising:
module M3.1.2.1: deleting rows continuously 0 from the first row of the device history data set;
Module M3.1.2.2: deleting a fixed proportion of rows from the last row of the historical data set of the device;
the standardized processing equipment history data z in the module M3.1.3 is calculated by the following steps:
wherein x is data in the equipment history data set, mu is the mean value of the equipment history data set, and s is the standard deviation of the equipment history data set;
the method of feature selection of the device history dataset in module M3.1.4 includes:
module M3.1.4.1: calculating mutual information of each feature of the historical data set of the equipment and the predicted target feature;
module M3.1.4.2: sorting the characteristics of the historical data set of the equipment according to the mutual information;
module M3.1.4.3: the feature with the lowest mutual information of fixed proportion is deleted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310619803.4A CN116647507A (en) | 2023-05-29 | 2023-05-29 | Internet of things data processing method and system based on load self-adaption |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310619803.4A CN116647507A (en) | 2023-05-29 | 2023-05-29 | Internet of things data processing method and system based on load self-adaption |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116647507A true CN116647507A (en) | 2023-08-25 |
Family
ID=87618318
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310619803.4A Pending CN116647507A (en) | 2023-05-29 | 2023-05-29 | Internet of things data processing method and system based on load self-adaption |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116647507A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117453759A (en) * | 2023-12-19 | 2024-01-26 | 深圳竹云科技股份有限公司 | Service data processing method, device, computer equipment and storage medium |
-
2023
- 2023-05-29 CN CN202310619803.4A patent/CN116647507A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117453759A (en) * | 2023-12-19 | 2024-01-26 | 深圳竹云科技股份有限公司 | Service data processing method, device, computer equipment and storage medium |
| CN117453759B (en) * | 2023-12-19 | 2024-04-02 | 深圳竹云科技股份有限公司 | Service data processing method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Manogaran et al. | Machine learning assisted information management scheme in service concentrated IoT | |
| CN107360032B (en) | Network flow identification method and electronic equipment | |
| CN116647507A (en) | Internet of things data processing method and system based on load self-adaption | |
| CN112860337B (en) | Method and system for unloading dependent tasks in multi-access edge computing | |
| CN111294812A (en) | Method and system for resource capacity expansion planning | |
| CN108833227B (en) | Intelligent home communication optimal scheduling system and method based on edge calculation | |
| CN115794407A (en) | Computing resource allocation method and device, electronic equipment and nonvolatile storage medium | |
| Kaur et al. | Dynamic resource allocation for big data streams based on data characteristics (5 V s) | |
| CN114513470B (en) | Network flow control method, device, equipment and computer readable storage medium | |
| CN114500578A (en) | Load balancing scheduling method and device for distributed storage system and storage medium | |
| CN110162390A (en) | A kind of method for allocating tasks and system of mist computing system | |
| CN111191113B (en) | Data resource demand prediction and adjustment method based on edge computing environment | |
| CN112559078B (en) | Method and system for hierarchically unloading tasks of mobile edge computing server | |
| KR20230032754A (en) | Apparatus and Method for Task Offloading of MEC-Based Wireless Network | |
| Ponmalar et al. | Machine Learning Based Network Traffic Predictive Analysis | |
| AbdulRahman et al. | Management of digital twin-driven IoT using federated learning | |
| CN113676357A (en) | Decision method for edge data processing in power internet of things and application thereof | |
| Sharara et al. | A recurrent neural network based approach for coordinating radio and computing resources allocation in cloud-ran | |
| Jang et al. | Fast quality driven selection of composite Web services | |
| Lee et al. | Machine learning and deep learning for throughput prediction | |
| Costa et al. | Intelligent resource sharing to enable quality of service for network clients: the trade-off between accuracy and complexity | |
| Ma et al. | Quality-aware video offloading in mobile edge computing: A data-driven two-stage stochastic optimization | |
| Adegboyega | Time-series models for cloud workload prediction: A comparison | |
| Zhou et al. | Study on the evolutionary optimisation of the topology of network control systems | |
| CN114077482B (en) | Intelligent computing optimization method for industrial intelligent manufacturing edge |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |