CN113452676B - Detector distribution method and Internet of things detection system - Google Patents
Detector distribution method and Internet of things detection system Download PDFInfo
- Publication number
- CN113452676B CN113452676B CN202110583804.9A CN202110583804A CN113452676B CN 113452676 B CN113452676 B CN 113452676B CN 202110583804 A CN202110583804 A CN 202110583804A CN 113452676 B CN113452676 B CN 113452676B
- Authority
- CN
- China
- Prior art keywords
- detector
- detection
- gateway
- edge
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y10/00—Economic sectors
- G16Y10/80—Homes; Buildings
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y40/00—IoT characterised by the purpose of the information processing
- G16Y40/10—Detection; Monitoring
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16Y—INFORMATION AND COMMUNICATION TECHNOLOGY SPECIALLY ADAPTED FOR THE INTERNET OF THINGS [IoT]
- G16Y40/00—IoT characterised by the purpose of the information processing
- G16Y40/50—Safety; Security of things, users, data or systems
Abstract
The invention discloses a detector allocation method and an Internet of things detection system, wherein the method comprises the following steps: receiving a detection request of an edge device; when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device, wherein the detection file comprises a transit address or a device detector corresponding to the gateway device; and sending the detection file to the edge equipment, and controlling the edge equipment to carry out network flow detection based on the detection file. The cloud end can send different detection files according to the type and the state of the current edge equipment, and improves the detection efficiency on the basis of ensuring the detection precision of the edge equipment.
Description
Technical Field
The invention relates to the field of Internet of things, in particular to a detector distribution method and an Internet of things detection system.
Background
Aiming at the deployment position of a safety detection system or a monitoring system of the Internet of things, the current deployment thinking is that detection equipment can be best placed at a position as close to an attack source as possible, so that the attack can be quickly detected, and timely and effective mitigation measures are taken before the attack causes large-scale damage.
In internet of things botnets, infected internet of things devices are attack sources, and these devices continuously receive commands and controls (C & C) from attackers and destroy other devices or services by using various means, such as Distributed Denial of Service (DDoS) attacks, scanning attacks, sending spam, leaking private data, and so on. Therefore, edge computing is a promising deployment method for mitigating network attacks using the internet of things, for example, some existing solutions deploy an intrusion detection scheme using a home router as an edge device to monitor traffic sent out by a terminal device in the smart home internet of things. However, on one hand, the resource expandability of the home router is poor, large-scale flow detection is difficult to realize in practical use by using a single router as an edge device, the detection efficiency is low, and effective detection cannot be achieved, and on the other hand, even if potential malicious behaviors of the terminal device can be detected, common consumers often lack sufficient safety awareness and professional knowledge and cannot make further effective measures on the infected device, such as authority management, vulnerability repair, firmware/software update, malicious IP block and the like. Therefore, the current detection device deployment scheme cannot achieve an effective effect in the field of home internet of things.
Disclosure of Invention
The invention mainly aims to provide a detector allocation method and an Internet of things detection system, and aims to solve the problem that in the prior art, effective detection effect cannot be achieved by the arrangement of detection equipment in a family Internet of things.
To achieve the above object, the present invention provides a detector allocation method, including the steps of:
receiving a detection request of an edge device;
when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device, wherein the detection file comprises a transit address or a device detector corresponding to the gateway device;
and sending the detection file to the edge equipment, and controlling the edge equipment to carry out network flow detection based on the detection file.
Optionally, the detector allocation method, wherein, when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device specifically includes:
when the edge device is a gateway device, determining a gateway state of the gateway device according to a preset load state table;
when the gateway state is an idle state, sending a device detector corresponding to the gateway device;
and when the gateway state is an overload state, determining transfer equipment corresponding to the gateway equipment according to the address of the gateway equipment, sending the equipment detector to the transfer equipment, and sending the equipment address of the transfer equipment serving as a transfer address to the gateway equipment.
Optionally, the detector allocation method, wherein the device detector comprises a raw detector and a compressed detector;
and performing integer quantization on the original detector to generate a compressed detector.
In addition, to achieve the above object, the present invention further provides a cloud, wherein the cloud includes: a memory, a processor and a detector distribution program stored on the memory and executable on the processor, the detector distribution program when executed by the processor implementing the steps of the detector allocation method as described above.
Further, to achieve the above object, the present invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a detector distribution program, and the detector distribution program realizes the steps of the detector allocation method as described above when executed by a processor.
In addition, in order to achieve the above object, the present invention further provides an internet of things detection system, wherein the internet of things detection system includes a plurality of terminal devices, edge devices, and the cloud end;
the terminal equipment is used for sending a connection request and network flow to the edge equipment;
the edge device comprises a request module and a detection module;
the request module is used for establishing communication connection with the terminal equipment and sending a detection request to the cloud when receiving a connection request sent by the terminal equipment;
the detection module is used for detecting the network flow of the terminal equipment based on a detection file when the detection file sent by the cloud end of the Internet of things is detected;
the cloud comprises a receiving module, a searching module and a sending module;
the receiving module is used for receiving a detection request of the edge device;
the searching module is configured to determine a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device when the edge device is the gateway device, where the detection file includes a transit address or a device detector corresponding to the gateway device; and the number of the first and second groups,
the sending module is used for sending the detection file to the edge device and controlling the edge device to carry out network flow detection based on the detection file.
Optionally, the internet of things detection system, wherein the search module includes:
the state module is used for determining the gateway state of the gateway equipment according to a preset load state table when the edge equipment is the gateway equipment;
a first determining module, configured to, when the gateway state is an idle state, use a device detector corresponding to the gateway device as a detection file corresponding to the gateway device;
and the second determining module is used for determining the transfer equipment corresponding to the gateway equipment according to the address of the gateway equipment when the gateway state is the overload state, sending the equipment detector to the transfer equipment, and taking the equipment address of the transfer equipment as a detection file corresponding to the gateway equipment.
Optionally, the internet of things detection system, wherein the gateway device includes:
the connection module is used for establishing communication connection with the transfer equipment according to the equipment address when the detection file is a transfer address;
the processing module is used for preprocessing the network flow to generate a packet sequence when the terminal equipment sends the network flow;
the transmission module is used for transmitting the packet sequence to the transfer equipment; and the number of the first and second groups,
when receiving the packet sequence, the relay device is configured to perform anomaly detection on the packet sequence based on the device detector.
Optionally, in the internet of things detection system, the sending module is further configured to send, when the edge device is an edge server, a device detector corresponding to the edge server as a detection file to the edge server;
the edge server includes:
the acquisition module is used for acquiring the adjusting flow in the network flow according to preset monitoring time when the network flow sent by the terminal equipment is received;
the adjusting module is used for adjusting parameters of the equipment detector based on the adjusting flow to obtain a fine-tuning detector; and the flow detection module is used for carrying out flow detection on the terminal equipment based on the fine adjustment detector.
Optionally, in the internet of things detection system, the device detector includes an auto-encoder, a full link layer, and an outlier detector;
the adjusting module is used for adjusting parameters of the full-connection stratum and the abnormal point detector according to the adjusting flow. The invention provides a detector allocation method based on a cloud side end system and an Internet of things detection system. The cloud end receives a detection request of the edge device, and if the edge device is a gateway device represented by an internet of things gateway, the computing power of the edge device is limited, so that a corresponding detection file needs to be selected according to the gateway state, namely the load state, of the current edge device. The detection file includes a transit address or a device detector corresponding to the gateway device. Therefore, when the state of the gateway device is idle, the gateway device can detect according to the obtained device detector; when the state of the gateway device is overloaded, that is, when the gateway device cannot perform more calculations, the network traffic detection is shared through the transit address and the transit device corresponding to the transit address, so that the detection efficiency is improved, and the attack on the abnormal internet of things device is effectively monitored.
Drawings
FIG. 1 is a schematic diagram of an IOT detection system in accordance with a preferred embodiment of the present invention;
FIG. 2 is a flow chart of a preferred embodiment of the detector allocation method of the present invention;
FIG. 3 is a diagram illustrating network traffic detection when an edge device is a gateway device according to a preferred embodiment of the detector allocation method of the present invention;
FIG. 4 is a schematic diagram of network traffic detection when an edge device is a gateway server according to the preferred embodiment of the detector allocation method of the present invention;
fig. 5 is a schematic view of an operating environment of a cloud according to a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the detector allocation method according to the preferred embodiment of the present invention, the detector allocation method can be executed through a cloud, and the cloud can be a server, a management platform, and the like. In this embodiment, the cloud and the detection system of the internet of things are collocated for work, so the description of the detector allocation method is performed by taking the workflow of the detection system of the internet of things as an example. As shown in fig. 1, the internet of things detection system includes a plurality of terminal devices, an edge device, and a cloud.
The terminal equipment is used for sending a connection request and network flow to the edge equipment;
the edge device comprises a request module and a detection module;
the cloud comprises a receiving module, a searching module and a sending module;
the request module is used for establishing communication connection with the terminal equipment and sending a detection request to a cloud end when receiving a connection request sent by the terminal equipment;
the detection module is used for detecting the network flow of the terminal equipment based on a detection file when the detection file sent by the cloud end of the Internet of things is detected;
the receiving module is used for receiving a detection request of the edge device;
the searching module is used for determining a detection file corresponding to the gateway equipment according to the gateway state corresponding to the gateway equipment when the edge equipment is the gateway equipment; and the number of the first and second groups,
the sending module is used for sending the detection file to the edge device and controlling the edge device to carry out network flow detection based on the detection file.
Specifically, the internet of things detection system comprises a cloud end, edge equipment and a plurality of terminal equipment. For each terminal device, the terminal device sends a connection request to the edge device before joining the internet of things. The edge device includes a request module and a detection module. The connection request is mainly used for making a request for joining the internet of things to the edge device. And when receiving a connection request sent by the terminal equipment, the request module establishes communication connection with the terminal equipment. And a joining rule can be preset, and only the terminal equipment which accords with the joining rule can join the Internet of things. After the terminal equipment joins the internet of things, the edge equipment captures the network traffic and detects the network traffic when the terminal equipment sends the network traffic to the outside every time.
In this embodiment, because the time for sending the network traffic of each terminal device is not consistent, a phenomenon of time cross overlapping may exist, and in order to effectively detect each terminal joining the internet of things in time, the request module establishes a communication connection with the terminal device and also sends a detection request to the cloud to obtain a detection file for detection.
According to different use scenes, two typical types of edge equipment serving as a cloud edge system exist in the internet of things: one type of gateway devices represented by internet of things gateways, such as home wireless routers, is weak in computing resources, and is often used in smart homes, smart offices and other usage scenarios; another type is an Edge server, such as an Edge host supported by an aristoloc Link IoT Edge or amazon AWS iotgrenggrass, which has more abundant resources and is often used in smart cities, industrial internet of things, enterprise internet of things, or larger-scale home internet of things. In order to adapt to the resource difference of the two different types of edge devices at the same time, the present embodiment may adopt two different operation modes according to the type of the edge device used.
In the application scenario of the internet of things network, a main challenge of using the gateway device to provide an anomaly detection service is resource shortage, especially for a detection system based on a machine learning algorithm. When the number of terminal devices in the internet of things network continuously increases, the gateway device will correspondingly request more detectors from the cloud, and the computing power of the gateway device is limited, so when more detectors exist, the processing efficiency will be reduced, further shortage of resources will result in processing delay, and adverse effects such as influencing the normal function of the gateway, and the cloud in this embodiment includes a receiving module, a searching module and a sending module, which are in communication connection with each other. Based on the cloud, the device detectors corresponding to the gateway devices may be distributed, as shown in fig. 2, the process includes:
s100, receiving a detection request of edge equipment;
s200, when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device, wherein the detection file comprises a transit address or a device detector corresponding to the gateway device;
s300, sending the detection file to the edge device, and controlling the edge device to perform network flow detection based on the detection file.
Specifically, the receiving module is configured to receive a detection request of an edge device. And when the edge device is the gateway device, the search module determines the detection file corresponding to the gateway device according to the gateway state corresponding to the gateway device.
In a first implementation manner of this embodiment, the detection request sent by the gateway device may include a gateway state obtained by evaluating the gateway device by the gateway device, and the search module directly determines the detection file corresponding to the gateway device according to the gateway state.
In a second implementation manner of this embodiment, as shown in fig. 3, a multi-gateway cooperation mode is adopted, and by cooperatively utilizing resources of multiple gateways, we can reduce the resource consumption pressure of a single gateway.
The searching module determines the gateway state corresponding to the gateway device and realizes the gateway state through the load state table. The gateway equipment firstly carries out resource evaluation on own resources, sets the maximum number of the equipment detectors which can be loaded on the basis of keeping the detection efficiency according to the evaluated resources, and records the maximum value as the expected number and the expected number
Wherein E1The number of the gateway device. When the gateway equipment obtains a detection file each time, the number of the current equipment detectors is calculated and is taken as the current number and recorded as the current number
Then the current quantity and the expected quantity are compared, and the load state of the gateway equipment is updated according to the sizes of the current quantity and the expected quantity. If it is
The gateway state of the current gateway device is indicated as idle state, that is, the number of device detectors can be increased in the gateway device, and the load state is set
Is updated to 1, which can be expressed as
If not, the gateway state of the current gateway device is an overload state, namely the gateway device is already overloaded, the detection efficiency is reduced due to the fact that the number of the device detectors is increased, and the load state is changed
Is updated to 0 and can be expressed as
And updating each pair of load states, and sending the load states to the cloud end by the gateway equipment. And the cloud end summarizes the device address of each gateway device and the corresponding load state into a load state table. And when the state unit in the searching module receives the detection request, determining the gateway state of the gateway equipment according to a preset load state table. When the value corresponding to the gateway equipment in the load state table is 0, determining that the gateway state of the gateway equipment is an overload state; and when the corresponding numerical value is 1, determining that the gateway state of the gateway equipment is an idle state. And after the gateway state is determined, the searching module determines the corresponding detection file according to the gateway state.
In this embodiment, if the gateway state is an idle state, a first determining unit in the lookup module directly sends a device detector corresponding to the gateway device as a detection file to the gateway device; if the gateway equipment is in the overload state, a second determining unit in the searching module searches corresponding transfer equipment according to the address of the gateway equipment, simultaneously sends an equipment detector to the transfer equipment, and sends the equipment address of the transfer equipment serving as a transfer address in a detection file to the gateway equipment. In a first implementation manner of this embodiment, the transit device may be a preset device dedicated to performing traffic detection on network traffic captured by the edge terminal. In a second implementation manner of this embodiment, the transit device is one of a plurality of edge devices connected to the cloud, and the cloud searches for a gateway device that is closer to the gateway device that sends the detection request and has an idle gateway state, from among the edge devices, through a load state value in the load state table and an address of each edge device, and uses this gateway device as the transit device corresponding to the gateway device that sends the detection request.
If the detection file is the device detector, the gateway device can detect the network traffic of the terminal device based on the device detector. If the detection file is a transfer address, the gateway device establishes communication connection with the transfer device according to the device address, and performs network traffic detection on network traffic externally sent by the terminal device based on a device detector of the transfer device, so as to achieve the effect of cooperative work of a plurality of gateway devices and improve the detection efficiency of network traffic detection which can be performed by the edge device.
Further, the device detector in this embodiment includes two types, one is a detector of a normal mode used directly after training, which is called a raw detector, and the other is a compressed detector obtained by compressing the raw detector. The compression detector can reduce the size of the detector on the basis of providing network flow, and reduce the occupation of resources. In this embodiment, the original detection is compressed by using integer quantization, that is, the original detector is subjected to integer quantization to generate a compressed detector.
Meanwhile, the device detector may be a detector capable of completely detecting the network traffic, or may be a detector including only a substantial network traffic detection. If the work flows of the network traffic detection corresponding to different terminal devices are relatively close, and the parts for preprocessing the network traffic can be uniformly executed through the edge device, the device detector can mainly perform the preprocessing and then perform the substantial network traffic detection. The adopted equipment detector can comprise an equipment detector realized by the existing methods such as the Internet of things edge botnet attack detection based on sparse representation and the like. In this embodiment, the gateway device includes a connection module, a processing module, and a transmission module, where the connection module is configured to establish a communication connection with the transit device according to the device address when the detection file is a transit address. The step of detecting the network flow is executed by an abnormality detection module, the abnormality detection module comprises a catcher, a preprocessor and a plurality of device detectors, and the processing module comprises the catcher and the preprocessor in the abnormality detection module. The processing module is used for preprocessing the network flow to generate a packet sequence when the terminal equipment sends the network flow. The capturer and the preprocessor may be provided in only one edge device, and all network traffic flowing through the edge device is captured by the capturer and input to the preprocessor.
The preprocessor is used for converting network traffic into a data form which can be input by a machine learning model. The data format convenient for calculation is mainly vector format, so vectorization is taken as an example for illustration. Botnet traffic tends to exhibit unique sequence characteristics, e.g., successive small packets may represent a C & C connection waiting to receive an attack order. Therefore, in this embodiment, the preprocessor converts a set of consecutive data packets into a sequence data sample, so as to better reveal the sequence relationship between the data packets.
The network flow consists of a plurality of data packets, and when the preprocessor receives the network flow, each data packet in the network flow is firstly vectorized to obtain a plurality of first vectors. And sequencing and integrating bytes of each data packet according to the sequence number of each data in the sequencing result to obtain a first vector. In this embodiment, the bytes for sorting and integrating are bits, and the value of each bit is between 0 and 255. Therefore, in the aspect of feature extraction, the preprocessor references the application of deep learning in computer vision, uses the original bytes of the data packet as features, and uses the deep learning to automatically extract potential features, thereby greatly reducing the dependence on feature engineering based on professional knowledge.
Since the overfitting problem is easy to occur in the process of proceeding and learning, the first vector needs to be partially modified, and the overfitting is reduced. And then, aiming at each first vector, modifying the first vector according to a preset modification rule to generate a second vector. The modification rule in this embodiment includes a plurality of preset rules for modifying the first vector, and in this embodiment, the modification rule includes a modification rule for modifying a local network configuration field in the first vector. Based on the modification rule, the data link layer is removed and the Internet Protocol (IP) address is zero-masked, i.e., the original IP address is replaced by 0 of the same length. In addition, considering that the machine learning model requires that the length of the input is fixed, in this embodiment, the modifying rule further includes padding the first vector. Each data packet is zero-padded to the length of the Maximum Transmission Unit (MTU), that is, 1500 bytes, to obtain a second vector, which ensures that the information content of the data in the data packet is completely reserved.
In order to obtain better convergence effect, the preprocessor normalizes the second vector to generate a third vector. In this embodiment, each byte in the second vector is divided by the maximum value 255 of the byte, and converted into a value between 0 and 1. To this end, a packet is converted into a third vector of 1500 dimensions, namely:
wherein, bn(n∈[1,d]) For each byte corresponding to the packet, T is transposed, bn∈[0,1]D, the dimension of the third vector of the preceding, is 1500.
And then the preprocessor generates a plurality of packet sequences corresponding to the network flow according to the third vector. In a first implementation manner of this embodiment, a third vector corresponding to network traffic originating from the same terminal device is directly used as a corresponding packet sequence for subsequent analysis.
There may be some similarity in data due to different network connections corresponding to different service functions. For example, a certain terminal device is a smart television, which includes functions of shooting, recording, and video stream acquisition, and connection ports corresponding to different functions may have differences, so that different connections are responsible for different activities. Thus, the pre-processing performs packet aggregation on packets from the same connection. For example, all data packets sent by the camera and the domain name "google.com" using the port 8080 of the smart television may be regarded as coming from the same connection, and then the data packets from the connection established with the port 8080 in the network traffic sent by the smart television are classified into one type. And buffering the aggregated data packets by adopting a sliding window, wherein each sliding window can be regarded as a matrix.
And aiming at each terminal device, setting a plurality of different first-in first-out blank matrixes according to different network connections. In the actual processing, the preprocessor processes the data packets sequentially according to the time sequence, so that when the data packets are acquired, for each data packet, the sample matrix corresponding to the data packet is determined according to the network connection corresponding to the data packet. When no data exists in the sample matrix, the sample matrix is a preset blank matrix. Filling the third vector into the sample matrix in sequence to generate a packet sequence corresponding to the network flow, i.e. the sample matrix is stored from the same connection ciAnd constitute a sequence data sample. If it is not
Filled, a packet sequence X is output as follows:
in a first implementation manner of this embodiment, after each filling, it is determined whether a sample matrix is filled, and if so, the filled sample matrix is used as a packet sequence corresponding to a network traffic; if not, the sample matrix continues to wait for a subsequent third vector until filled.
This process can be viewed as the sample matrix sampling at intervals in units of matrix length over a sequence of sequentially input third vectors. In a second implementation manner of this embodiment, a blank sample matrix is filled first, and when a third vector is obtained every time, the third vector filled first in the sample matrix is excluded, and a newly obtained sample matrix is written to obtain a new packet sequence. This process can be viewed as a sampling of the sample matrix in units of a single vector over a sequence of sequentially input third vectors. In addition, the first embodiment and the second embodiment have little difference in detection effect, so this embodiment will take the first embodiment as an example for the following description.
It should be noted that the number of rows in the sample matrix in the embodiment is the dimension d, and the number of columns can be randomly set. In consideration of the difference in functions, the terminal devices may be classified into stream data type devices such as smart televisions and cameras, and control type devices such as smart lights and WiFi sockets. Generally, the data stream length of the stream data type device is longer, and the data stream length of the control type device is shorter. For the sequence model, the longer the sequence length is, the larger the amount of information contained, but the greater the difficulty of learning. Therefore, in order to accurately identify the terminal devices of different device types, the information amount and the training difficulty need to be balanced, and the deviation of the accuracy needs to be reduced. In this embodiment, before setting the blank matrix, the preprocessor acquires the device types of the plurality of terminal devices and the network data of each terminal device, where the device types include a stream data class and a control class. Five tuple flows from the same device type are sampled and the number of flow samples is calculated. And then calculating the matrix length corresponding to the equipment type according to the quintuple flow and the flow sample number. The calculation method adopted in this embodiment is to determine the matrix length according to a preset window length formula. The window length formula is:
wherein f isiRepresenting a sampled five tuple flow, k representing the number of flow samples. And after the matrix length is obtained through calculation, the matrix length is used as the column number of the sample matrix, the vector dimension is used as the row number of the sample matrix, and the size of the sample matrix is determined.
If the gateway state of the current gateway device is an idle state, the cloud sends the device detector to the gateway device, and the gateway device directly detects the packet sequence obtained by preprocessing by using the device detector. And if the gateway state of the current gateway equipment is an overload state, a transmission module in the gateway equipment sends the packet sequence to the transfer equipment.
In order to improve the detection efficiency of the packet sequence, the device detector in this embodiment includes an auto-encoder and an outlier detector, where the auto-encoder may adopt a shrinking auto-encoder, a regular auto-encoder, and the auto-encoder adopted in this embodiment is a long and short memory auto-encoder. The abnormal point detector comprises an abnormal point detection algorithm, and can detect the abnormal points of the data coded by the self-coder.
The packet sequence is input to the trained self-encoder in this embodiment, which is a long-short term memory self-encoder. Long-short term memory self-coding is an unsupervised model that can transform an incoming sequence of packets into another sequence. An autoencoder needs to encode during trainingThe device, decoder and full connection layer are jointly trained. A large amount of data obtained by performing the above preprocessing process on normal data is collected as training data, the normal data in this embodiment refers to transmission data in a general state, and relatively speaking, the abnormal data is transmission data for an attack, such as zombie data. The encoder first compresses the input training data into a low-latitude latent implicit expression, and for the input training data
Each one of which is
The following transformations are followed:
ct＝ft·ct-1+it·lt；
ht＝ot·tanh(ct)。
in the long-short term memory self-coding model, sequences are processed in sequence, so that t represents the time of the time sequence, and t-1 represents the output value of the network at the last time; c represents the state of the cell at a certain time; []Show two willConnecting the vectors into a longer vector; σ represents an activation function sigmoid; tanh represents an activation function tanh; the long-short term memory self-coding model consists of three gates, wherein each gate is a weight matrix WqAnd bias term bq(q∈[l,o,i]) And forming a full connection layer. The three gates are a forgetting gate (forgetgate), an input gate (inputgate) and an output gate (output gate), respectively. Forgetting the door determines the unit state c of the last momentt-1How much information, W, is kept until the current time, and can be stored for a long timefWeight matrix representing forgetting gate, bfIs a biased term for a forgetting gate; the input gate determines the input of the network at the current moment
How many cells to save to cell state ctThe current irrelevant content input memory, W, can be avoidediWeight matrix representing input gates, biIs the offset term of the input gate; the output gate determines the cell state ctHow much output value h is output to the LSTM current timet，WoWeight matrix representing output gates, boIs the bias term for the output gate. ltFor describing the state of the currently input cell, using the forgetting gate value ftMultiplying by the cell state at the previous moment ct-1Reuse the input threshold value itMultiplied by ltAdding the two to obtain the unit state c at the current timetReuse of the output threshold value otMultiplying by ctThe tan h activation value can obtain the output value h of the current momentt。htI.e. an implicit expression that the timestamp is t. The decoder then decodes the implicit table using the same number of neurons as the encoder, resulting in a high dimensional representation. Finally, the full-link layer maps the expression into a result sequence with the same length and dimension as the input training sequence, and the result sequence is recorded as
And then calculating the difference between the result sequence and the training sequence according to a preset loss function to obtain the training error between the result sequence and the training sequence. In this embodiment, the way to calculate the training Error between the two is Root Mean Square Error (RMSE), and the formula is as follows:
RMSE can be interpreted as the error resulting from reconstruction of the training sequence and then transmitted back to the self-encoder based on the error value, adjusting the parameters of the model to achieve minimization of the difference between the training sequence and the resulting sequence.
After the encoder and decoder are trained, since the training data used in the self-encoder training process is normal data, the error value obtained from abnormal data may deviate from the error value obtained from normal data in the inference stage, so that an abnormality may be detected.
And inputting the packet sequence obtained in the foregoing into a trained self-encoder, and encoding and decoding the packet sequence by the self-encoder as the same as the data to be trained so as to reconstruct the packet sequence and generate a reconstructed sequence corresponding to the packet sequence.
In a first implementation manner of this embodiment, a reconstruction error threshold is preset, after a reconstruction sequence corresponding to a packet sequence is obtained, an error between the packet sequence and the reconstruction sequence is calculated according to a loss function adopted during training of a self-encoder to obtain a reconstruction error, and then whether the reconstruction error is within a reconstruction error threshold range is determined, so as to determine whether the packet sequence is an abnormal sequence.
In a second implementation manner of this embodiment, in a process of training a self-encoder, the training error is input into the abnormal point detection model, and an error threshold corresponding to the training error is calculated by the abnormal point detection model. Through comparison and realization of various models, the embodiment uses a single-class support vector machine in a singularity evaluation algorithm as an optimal outlier detector, instead of an outlier detection algorithm. The single-class support vector machine is a typical high-efficiency novel singularity detection algorithm. Therefore, when the reconstruction error is inputted to the abnormal point detector, the abnormal point detector may determine whether the reconstruction error is less than or equal to the error threshold. If so, determining that the traffic type corresponding to the packet sequence is normal traffic; if not, determining that the traffic type corresponding to the packet sequence is abnormal traffic.
After the sequence type corresponding to each packet sequence is obtained, the packet sequence corresponds to a data packet in the network traffic, so that the judgment of the sequence type of the packet sequence can be regarded as the judgment of the distribution condition of a normal data packet in the network traffic, and the traffic type of the network traffic can be further determined.
For example, if the network connection corresponding to the multiple packet sequences is network connection a and the corresponding sequence types are all abnormal sequences, the data packet transmitted with network connection a is determined as abnormal traffic data, so that the network traffic type of the terminal device is traffic abnormality, and the terminal device may be an abnormal device, that is, an attacked device or an attacking device.
And then, the edge gateway generates a detection report corresponding to the terminal equipment according to the detection result and sends the detection report to the management platform. The detection report can be sent at regular time, or when the detection result is abnormal equipment, the detection report is sent to the management platform. That is, if the transfer device completes the detection result of a certain terminal device, the transfer device sends a detection report corresponding to the terminal device.
Based on the device detector, the original detector in this embodiment is the trained self-encoder and outlier detector, and the compressed detector is the detector that performs the integer quantization on the self-encoder. The main mode is to convert the 32-bit floating point type weight in the long-short term memory self-encoder into 8-bit integer type weight by using the quantization technology after training, thereby reducing the size of the long-short term memory self-encoder by 75 percent and realizing twice acceleration of the reasoning process. The accuracy of the compressed detector was tested to be very slightly degraded compared to the original detector.
For large internet of things networks, the edge device may also be some kind of edge server deployed in a close location, such as a personal computer, a local server, or an edge cloud. Edge servers typically have more abundant computing resources to support more complex processes than gateway devices. In a large-scale internet of things network, due to the fact that the number of terminal devices is large, behavior modes of different terminal devices have better relevance with local environments and user preferences. For example, some users prefer to use apps to control a Wi-Fi camera, while some users prefer to use voice assistants to interact with it. In this case, even the same type of terminal device may present widely different traffic characteristics, and the same trained device detector may not contain all the characteristics in different use cases, so the false alarm rate may increase.
In this embodiment, as shown in fig. 4, an adaptive mode is adopted to detect network traffic of the edge server. The adaptive mode is mainly through fine tuning of the device detector to become a specific detector more adaptive to the local environment and user preferences. Retraining the device detector on the edge device is feasible due to the more abundant computing power of the edge server. The device detector uses the network flow sent by the terminal device in the current environment to retrain, so that the current use preference can be better subjected to active side writing, and the detector is more suitable for the current environment. The edge server comprises an acquisition module, an adjustment module and a flow detection module.
Firstly, when the edge device is an edge server, the sending module of the cloud further sends a device detector corresponding to the edge server as a detection file to the edge server. When the edge server receives the network traffic sent by the terminal equipment, the network traffic within a certain time interval is collected as the adjusting traffic. After most terminal devices are connected to the internet of things, certain preference changes occur to traffic behaviors, but the traffic behaviors tend to be stable after a period of time, and the traffic behaviors can be used as normal network traffic. Different terminal devices have different time for the device types to be stable, for example, the behavior of the voice assistant with stronger interactivity is more complicated, so different monitoring time is set for different types of terminal devices in advance, and the network flow is capturedDuring measurement, network traffic is collected as regulated traffic for regulating the plant detector based on this monitoring time
. And then adjusting the flow based on the flow data, and adjusting parameters of the equipment detector to obtain a fine adjustment detector. The edge server comprises an adjusting module, and the adjusting module is used for finely adjusting the equipment detector.
In the first implementation of this embodiment, all components in the device detector are included in the self-encoder and the outlier detector. Although the fine tuning effect is better, the method has more computing resources and takes longer time. In a second implementation, the only objects to be trimmed are the full-link hierarchy and the outlier detector. On one hand, because the main task of the fully-connected hierarchical and outlier detector is to extract potential features, the process is basically consistent for the same type of equipment; on the other hand, more computing resources are needed for training the two layers, the time consumption is longer, and the retraining time can be greatly reduced by only training the fully-connected layer and the abnormal point detector, so that the possibility that the equipment is infected in the retraining process is reduced. Through experiments, the device detector is retrained by using a small amount of adjusting flow sampled from the local environment and the preference of a user, and the false alarm rate of the fine-tuning detector obtained after adjustment can be greatly reduced. And after the fine tuning detector is obtained, carrying out flow detection on the terminal equipment based on the fine tuning detector.
Further, as shown in fig. 5, based on the above detector allocation method, the present invention further provides a cloud, where the cloud includes a processor 10, a memory 20, and a display 30. Fig. 5 shows only some of the components of the cloud, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
The storage 20 may be an internal storage unit of the cloud in some embodiments, such as a hard disk or a memory of the cloud. In other embodiments, the memory 20 may also be an external storage device of the cloud, such as a plug-in hard disk equipped on the cloud, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. Further, the memory 20 may also include both an internal storage unit of the cloud and an external storage device. The memory 20 is used for storing application software installed in the cloud and various data, such as program codes of the installation cloud. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a detector distribution program 40, and the detector distribution program 40 can be executed by the processor 10 to implement the detector allocation method of the present application.
The processor 10 may be, in some embodiments, a Central Processing Unit (CPU), microprocessor or other data Processing chip for running program codes stored in the memory 20 or Processing data, such as executing the detector allocation method.
The display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch panel, or the like in some embodiments. The display 30 is used for displaying information in the cloud and for displaying a visual user interface. The cloud components 10-30 communicate with each other via a system bus.
In one embodiment, the following steps are implemented when processor 10 executes detector distribution program 40 in memory 20:
receiving a detection request of an edge device;
when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device, wherein the detection file comprises a transit address or a device detector corresponding to the gateway device;
and sending the detection file to the edge equipment, and controlling the edge equipment to carry out network flow detection based on the detection file.
The present invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a detector distribution program, which when executed by a processor implements the steps of the detector allocation method as described above.
Of course, it will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The computer readable storage medium may be a memory, a magnetic disk, an optical disk, etc.
It is to be understood that the invention is not limited to the examples described above, but that modifications and variations may be effected thereto by those of ordinary skill in the art in light of the foregoing description, and that all such modifications and variations are intended to be within the scope of the invention as defined by the appended claims.
Claims (9)
1. A detector allocation method, characterized in that the detector allocation method comprises:
receiving a detection request of an edge device;
when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device, wherein the detection file comprises a transit address or a device detector corresponding to the gateway device;
when the edge device is a gateway device, determining a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device, specifically including:
when the edge device is a gateway device, determining a gateway state of the gateway device according to a preset load state table;
when the gateway state is an idle state, sending a device detector corresponding to the gateway device;
when the gateway state is an overload state, determining a transfer device corresponding to the gateway device according to the address of the gateway device, sending the device detector to the transfer device, and sending the device address of the transfer device serving as a transfer address to the gateway device;
and sending the detection file to the edge equipment, and controlling the edge equipment to carry out network flow detection based on the detection file.
2. The detector allocation method according to claim 1, wherein the device detector comprises a raw detector and a compressed detector;
and performing integer quantization on the original detector to generate a compressed detector.
3. A cloud, the cloud comprising: memory, a processor and a detector distribution program stored on the memory and executable on the processor, the detector distribution program when executed by the processor implementing the steps of the detector allocation method according to any one of claims 1-2.
4. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a detector distribution program which, when executed by a processor, implements the steps of the detector allocation method according to any one of claims 1-2.
5. An internet of things detection system, characterized in that the internet of things detection system comprises a plurality of terminal devices, edge devices and the cloud end of claim 3;
the terminal equipment is used for sending a connection request and network flow to the edge equipment;
the edge device comprises a request module and a detection module;
the request module is used for establishing communication connection with the terminal equipment and sending a detection request to the cloud when receiving a connection request sent by the terminal equipment;
the detection module is used for detecting the network flow of the terminal equipment based on a detection file when the detection file sent by the cloud end of the Internet of things is detected;
the cloud comprises a receiving module, a searching module and a sending module;
the receiving module is used for receiving a detection request of the edge device;
the searching module is configured to determine a detection file corresponding to the gateway device according to a gateway state corresponding to the gateway device when the edge device is the gateway device, where the detection file includes a transit address or a device detector corresponding to the gateway device; and the number of the first and second groups,
the sending module is used for sending the detection file to the edge device and controlling the edge device to carry out network flow detection based on the detection file.
6. The IOT detection system of claim 5, wherein the lookup module comprises:
a state unit, configured to determine, when the edge device is a gateway device, a gateway state of the gateway device according to a preset load state table;
a first determining unit, configured to, when the gateway state is an idle state, use a device detector corresponding to the gateway device as a detection file corresponding to the gateway device;
and a second determining unit, configured to determine, when the gateway status is an overload status, a transfer device corresponding to the gateway device according to the address of the gateway device, send the device detector to the transfer device, and use the device address of the transfer device as a detection file corresponding to the gateway device.
7. The internet of things detection system of claim 6, wherein the gateway device comprises:
the connection module is used for establishing communication connection with the transfer equipment according to the equipment address when the detection file is a transfer address;
the processing module is used for preprocessing the network flow to generate a packet sequence when the terminal equipment sends the network flow;
the transmission module is used for transmitting the packet sequence to the transfer equipment; and the number of the first and second groups,
when receiving the packet sequence, the relay device is configured to perform anomaly detection on the packet sequence based on the device detector.
8. The internet of things detection system of claim 6, wherein the sending module is further configured to send, when the edge device is an edge server, a device detector corresponding to the edge server as a detection file to the edge server;
the edge server includes:
the acquisition module is used for acquiring the adjusting flow in the network flow according to preset monitoring time when the network flow sent by the terminal equipment is received;
the adjusting module is used for adjusting parameters of the equipment detector based on the adjusting flow to obtain a fine-tuning detector;
and the flow detection module is used for carrying out flow detection on the terminal equipment based on the fine adjustment detector.
9. The internet of things detection system of claim 8, wherein the device detector comprises a self-encoder, a fully-connected hierarchy layer, and an outlier detector;
and the adjusting module is used for adjusting parameters of the full-link hierarchy and the abnormal point detector according to the adjusting flow.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110583804.9A CN113452676B (en) | 2021-05-27 | 2021-05-27 | Detector distribution method and Internet of things detection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110583804.9A CN113452676B (en) | 2021-05-27 | 2021-05-27 | Detector distribution method and Internet of things detection system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113452676A CN113452676A (en) | 2021-09-28 |
| CN113452676B true CN113452676B (en) | 2022-05-10 |
Family
ID=77810492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110583804.9A Active CN113452676B (en) | 2021-05-27 | 2021-05-27 | Detector distribution method and Internet of things detection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113452676B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114520736B (en) * | 2022-01-24 | 2023-08-22 | 广东工业大学 | Internet of things security detection method, device, equipment and storage medium |
| CN114448830B (en) * | 2022-03-07 | 2024-04-05 | 中国农业银行股份有限公司 | Equipment detection system and method |
| CN114826963B (en) * | 2022-03-31 | 2023-07-14 | 鹏城实验室 | Internet of things equipment detection method and system based on equipment behaviors |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103166926A (en) * | 2011-12-14 | 2013-06-19 | 中国科学院沈阳计算技术研究所有限公司 | Session initiation protocol (SIP) distributed denial of service (DDoS) attack distributed defensive system and load balancing method thereof |
| WO2017113273A1 (en) * | 2015-12-31 | 2017-07-06 | 华为技术有限公司 | Software defined data center and scheduling and traffic-monitoring method for service cluster therein |
| CN111817917A (en) * | 2020-07-03 | 2020-10-23 | 中移（杭州）信息技术有限公司 | Deep packet inspection method, device, server and storage medium |
| CN111866040A (en) * | 2019-04-28 | 2020-10-30 | 深圳长城开发科技股份有限公司 | Gateway load balancing method and device for LoRa server |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110611590A (en) * | 2019-09-16 | 2019-12-24 | 南京国电南自电网自动化有限公司 | Method and system for internet of things gateway communication backup |
| US11153350B2 (en) * | 2019-09-16 | 2021-10-19 | Fortinet, Inc. | Determining on-net/off-net status of a client device |
-
2021
- 2021-05-27 CN CN202110583804.9A patent/CN113452676B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103166926A (en) * | 2011-12-14 | 2013-06-19 | 中国科学院沈阳计算技术研究所有限公司 | Session initiation protocol (SIP) distributed denial of service (DDoS) attack distributed defensive system and load balancing method thereof |
| WO2017113273A1 (en) * | 2015-12-31 | 2017-07-06 | 华为技术有限公司 | Software defined data center and scheduling and traffic-monitoring method for service cluster therein |
| CN111866040A (en) * | 2019-04-28 | 2020-10-30 | 深圳长城开发科技股份有限公司 | Gateway load balancing method and device for LoRa server |
| CN111817917A (en) * | 2020-07-03 | 2020-10-23 | 中移（杭州）信息技术有限公司 | Deep packet inspection method, device, server and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113452676A (en) | 2021-09-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113364752B (en) | Flow abnormity detection method, detection equipment and computer readable storage medium | |
| CN113452676B (en) | Detector distribution method and Internet of things detection system | |
| CN111935170B (en) | Network abnormal flow detection method, device and equipment | |
| US8402543B1 (en) | Machine learning based botnet detection with dynamic adaptation | |
| WO2018054342A1 (en) | Method and system for classifying network data stream | |
| US20170374090A1 (en) | Using a machine learning classifier to assign a data retention priority for network forensics and retrospective detection | |
| US11516311B2 (en) | Distributed machine-learning resource sharing and request routing | |
| CN111064678A (en) | Network traffic classification method based on lightweight convolutional neural network | |
| CN110417729B (en) | Service and application classification method and system for encrypted traffic | |
| CN112804253B (en) | Network flow classification detection method, system and storage medium | |
| Ma | Analysis of anomaly detection method for Internet of things based on deep learning | |
| US11368482B2 (en) | Threat detection system for mobile communication system, and global device and local device thereof | |
| US20220303198A1 (en) | Method and apparatus for detecting anomaly of traffic of internet of things device based on automata | |
| Kanzaki et al. | Video streaming schemes for industrial IoT | |
| US20190124094A1 (en) | Active prioritization of investigation targets in network security | |
| Dvir et al. | Clustering the unknown-the youtube case | |
| Zhai et al. | Detection of TCP covert channel based on Markov model | |
| CN105099799A (en) | Botnet detection method and controller | |
| CN111291078A (en) | Domain name matching detection method and device | |
| CN115499230A (en) | Network attack detection method and device, equipment and storage medium | |
| Dener et al. | Rfse-gru: Data balanced classification model for mobile encrypted traffic in big data environment | |
| CN114866310A (en) | Malicious encrypted flow detection method, terminal equipment and storage medium | |
| CN115334005A (en) | Encrypted flow identification method based on pruning convolution neural network and machine learning | |
| Okui et al. | Identification of an iot device model in the home domain using ipfix records | |
| CN112468509A (en) | Deep learning technology-based automatic flow data detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |