CN110086860B - Data anomaly detection method and device under Internet of things big data environment - Google Patents

Abstract

The invention discloses a data anomaly detection method and device in an Internet of things big data environment. Then, all the context attributes are reduced and combined to form a corresponding table of the neighborhood sharing context and the equipment context, and a corresponding probability matrix is filled. And comprehensively analyzing all the devices in the same neighborhood in the detection process, calculating the possibility of sharing the context of each neighborhood according to the corresponding table, obtaining a final judgment result, and finally loading the probability matrix of each device according to the judged context and carrying out anomaly detection by using a probability detector algorithm. The method can detect the abnormal events lasting for a period of time, can adapt to the Internet of things equipment with various behavior modes, can solve the problem that the existing context judgment process is not credible by introducing neighborhood attributes, and improves the detection accuracy.

Description

A data anomaly detection method and device in the big data environment of the Internet of Things

technical field

The invention relates to the technical field of information security, in particular to a data abnormality detection method and device in the big data environment of the Internet of Things.

Background technique

The Internet of Things is a system that realizes the interconnection of people and things. With the rapid development of network technology, the Internet of Things system has been applied to various infrastructures to provide a wide range of services for the society. In recent years, a large number of new IoT big data analysis platforms have emerged that use the Internet of Things as the data source, big data as the analysis object, and artificial intelligence as the technical means. This requires the source data generated by the Internet of Things to have high credibility, otherwise it will affect the accuracy of subsequent big data analysis results and cause serious consequences.

In the prior art, commonly used IoT big data anomaly detection technologies mainly have three modes: Markov anomaly detector, anomaly detection technology based on sliding window, and context-aware anomaly detection technology.

In the process of implementing the present invention, the inventor of the present application found that the method of the prior art has at least the following technical problems:

The Markov anomaly detector is based on the Markov state transition matrix to detect the anomaly of the device. It considers that the current state is only related to the previous state and has nothing to do with any previous state. The process of training the model of Markov anomaly detector is simple, but the computational complexity of the detection process is high, and its characteristics make it only focus on abnormal mutations at a certain moment, and the problems of IoT data are often abnormal events that last for a period of time. , which will result in a lower detection accuracy.

After introducing the concept of sliding windows, a large number of researchers have proposed some anomaly detection methods based on sliding windows for different devices. The sliding window can pay attention to the behavior information of the IoT data for a period of time, can effectively deal with the abnormal situation of the data that lasts for a period of time, and improve the detection accuracy to a certain extent. However, such methods can only be aimed at relatively simple embedded devices, and the functions of IoT devices are becoming more and more complex, and there are various behavior modes. The traditional anomaly detection methods based on sliding windows are accurate in detecting such IoT devices. It will drop significantly and cannot cope with the increasingly powerful IoT scenarios.

In order to detect IoT devices with multiple behavior patterns, some methods have improved the sliding window anomaly detection technology and added a context detection module. Load different anomaly detection models for detection according to different context attributes. Although context-aware anomaly detection technology can detect anomalies of IoT data with contextual attributes to a certain extent, the current methods are all for the detection of a single device, and use the IoT data itself to judge the contextual attributes. If it is unreliable, the entire detection process will be unreliable, resulting in low accuracy of the detection results.

From this, it can be seen that the methods in the prior art have a technical problem of low accuracy.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a data anomaly detection method and device in the big data environment of the Internet of Things to solve or at least partially solve the technical problem of low accuracy in the methods in the prior art.

A first aspect of the present invention provides a data anomaly detection method under the big data environment of the Internet of Things, including:

Step S1: classify the IoT devices to be detected, and define a context attribute for each type of device, wherein each context attribute corresponds to a behavior mode of the corresponding type of device;

Step S2: Collect the IoT data generated when each type of device runs in each context attribute, and use a preset probability matrix algorithm to calculate the probability matrix of the context attribute corresponding to each type of device, so as to extract the characteristics of each type of device;

Step S3: according to all context attributes of all types of devices, obtain the context shared by all devices within the neighborhood scope, and use it as the neighborhood shared context;

Step S4: forming a neighborhood-device context correspondence table according to the context shared by all devices in the neighborhood, wherein the correspondence table includes a probability matrix;

Step S5: Calculate the possibility that the neighborhood where the device to be detected is located is in each neighborhood shared context, and determine the target neighborhood shared context corresponding to the neighborhood based on the calculated possibility;

Step S6: according to the determined target neighborhood shared context, load a probability matrix corresponding to the device to be detected from the neighborhood-device context correspondence table;

Step S7: Based on the loaded probability matrix, a preset probability matrix algorithm is used to perform anomaly detection on the data of each device to be detected.

In one embodiment, step S2 specifically includes:

Separate and independently train different contexts of different types of devices;

The normal operation data of all situations is collected by the data collector, and divided into no more than 10 segments according to its value range, wherein the symbol representing the segment to which the data belongs is used to represent a data;

The collected data is formed into a sequence with time as the dimension, and converted into a sequence of symbols, and a sliding window W with a fixed size n is defined to make it move in the direction of time flow;

At each moment, there is a character sequence of length n in the sliding window, count the number of double-character pairs with distances from 1 to n-1, and create a kind of double-character behavior with adjacent distances from 1 to 1. n-1 is the feature matrix of the column, and the number of statistics of the sliding window during the moving process is recorded in the feature matrix;

The matrix obtained by the above calculation method is normalized by column, and the probability of each character pair appearing under this distance is obtained, which is used as the corresponding probability matrix.

In one embodiment, step S3 specifically includes:

Comprehensively analyze all the context attributes, directly combine the unrelated contexts, reduce the repeated parts of the related contexts and then combine them to form a neighborhood shared context that includes all the contexts of all devices.

In one embodiment, step S5 specifically includes:

According to the neighborhood-device context correspondence table, a preset probability matrix algorithm is used to calculate the possibility that the neighborhood where the device to be detected is located is in the shared context of each neighborhood;

And take the most likely context as the target neighborhood to share the context.

In one embodiment, the calculation method of the possibility that the neighborhood where the device to be detected is located is in the shared context of each neighborhood includes:

min{P(i)}

P(i)=a*D(M _A ,S _Ai )+b*D(M _B ,S _Bi )+c*D(M _C ,S _Ci )+…

Among them, i represents the neighborhood sharing context number, a, b, c represent the number of devices A, B, and C in the neighborhood, D represents the function to calculate the Euclidean distance between two matrices, and S _Ai represents the neighborhood sharing context The probability matrix of device A corresponding to attribute i, S _Bi the probability matrix of device B corresponding to the neighborhood shared context attribute i, S _Ci represents the probability matrix of device C corresponding to the neighborhood shared context attribute i, M _A represents all A-type devices The average value of the probability matrix of , M _B represents the average value of the probability matrix of all B-type devices, and M _C represents the average value of the probability matrix of all C-type devices.

In one embodiment, step S7 specifically includes:

After the data of the device to be detected is converted into a character sequence, a sliding window W of equal size n is defined, and the probability of occurrence of the character sequence in the sliding window is calculated;

The calculated probability value is compared with the set threshold p. If it is smaller than the threshold value, it is marked as abnormal, and the window continues to slide. If k abnormal moments appear continuously, the data at this moment is detected as abnormal.

In one embodiment, when performing anomaly detection, the step of determining the shared context of the target neighborhood is performed every preset period.

Based on the same inventive concept, the second aspect of the present invention provides a data anomaly detection device in the big data environment of the Internet of Things, including:

The context attribute definition module is used to classify the IoT devices to be detected, and define a context attribute for each type of device, wherein each context attribute corresponds to a behavior mode of the corresponding type of device;

The device feature extraction module is used to collect the IoT data generated when each type of device runs in each context attribute, and use the preset probability matrix algorithm to calculate the probability matrix of the context attribute corresponding to each device to extract each type of device. characteristics of the device;

The neighborhood sharing context obtaining module is used to obtain the context shared by all devices in the neighborhood according to all the context attributes of all kinds of devices, and use it as the neighborhood sharing context;

a correspondence table forming module, configured to form a neighborhood-device context correspondence table according to the context shared by all devices in the neighborhood, wherein the correspondence table includes a probability matrix;

The target neighborhood shared context determination module is used to calculate the possibility that the neighborhood where the device to be detected is located is in each neighborhood shared context, and based on the calculated possibility, determine the target neighborhood shared corresponding to the neighborhood context;

The probability matrix loading module is used to load the probability matrix corresponding to the device to be detected from the neighborhood-device context correspondence table according to the determined target neighborhood sharing context;

The anomaly detection module is used to perform anomaly detection on the data of each device to be detected by using a preset probability matrix algorithm based on the loaded probability matrix.

Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed, the method described in the first aspect is implemented.

Based on the same inventive concept, a fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implements the following when executing the program. The method described in the first aspect.

The above-mentioned one or more technical solutions in the embodiments of the present application have at least one or more of the following technical effects:

The invention provides a data abnormality detection method under the big data environment of the Internet of Things. First, the Internet of Things devices to be detected are classified, and context attributes are defined for each type of equipment; The generated IoT data uses the preset probability matrix algorithm to calculate the probability matrix of the context attributes corresponding to each device; then, according to all the context attributes of all types of devices, the context shared by all devices in the neighborhood is obtained; then According to the context shared by all devices in the neighborhood, a neighborhood-device context correspondence table is formed; next, the possibility that the neighborhood where the device to be detected is located is in the shared context of each neighborhood is calculated, and based on the calculated possibility Determine the target neighborhood sharing context corresponding to the neighborhood, and then load the probability matrix corresponding to the device to be detected from the neighborhood-device context correspondence table according to the determined target neighborhood sharing context; finally, based on the loading The probability matrix of the device uses a preset probability matrix algorithm to perform abnormal detection on the data of each device to be detected.

Compared with the existing methods, in the process of detecting abnormal behavior of data in the Internet of Things, the present invention firstly determines in which context the current data of the device is generated, and after determining the context, from the pre-built neighborhood-device context. The probability matrix corresponding to the device and the context is selected from the correspondence table, that is, the present invention is a reliability determination method based on the neighborhood sharing context and data behavior, and can determine the context within the neighborhood range of multiple types of devices. It can detect data generated by IoT devices with various behavior patterns, and has the characteristics of high abnormal recognition rate, high detection accuracy rate, and low computational complexity in the detection process. The technical problem of low accuracy in the method in the prior art is solved.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

Fig. 1 is a flow chart of a data anomaly detection method in a big data environment of the Internet of Things according to the present invention;

2 is a flow chart of constructing a context-aware anomaly detection model according to an embodiment of the present invention;

3 is a schematic diagram of a flowchart of IoT big data anomaly detection based on neighborhood shared context attributes in a specific example;

FIG. 4 is a structural block diagram of a data anomaly detection device in an IoT big data environment according to an embodiment of the present invention;

FIG. 5 is a framework diagram of an IoT big data anomaly detection system according to an embodiment of the present invention;

6 is a structural diagram of a computer-readable storage medium in an embodiment of the present invention;

FIG. 7 is a structural diagram of a computer device in an embodiment of the present invention.

Detailed ways

The purpose of the present invention is to provide a data anomaly detection method and device in the big data environment of the Internet of Things, which aims to solve the problem that the perception layer of the Internet of Things is vulnerable to attacks and failures under the big data architecture of the Internet of Things, and the big data analysis center analyzes the error. Data giving wrong results can have serious consequences.

In order to achieve the above object, the present invention provides a method for detecting anomaly in IoT device data based on neighborhood shared context attributes, which can determine the context attributes within the neighborhood range of multiple types of devices, and can detect objects with multiple behavior patterns. The detection of data generated by networked devices has the characteristics of high abnormal recognition rate, high detection accuracy rate, and low computational complexity in the detection process.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Example 1

This embodiment provides a data anomaly detection method in an IoT big data environment, as shown in FIG. 1 , the method includes:

Step S1: classify the IoT devices to be detected, and define a context attribute for each type of device, wherein each context attribute corresponds to a behavior mode of the corresponding type of device.

Specifically, since each type of IoT device needs to be operated individually, the IoT devices to be detected are classified. And define context attributes for each type of device. For example, if a certain type of device has multiple behaviors, it has multiple context attributes.

Step S2: Collect IoT data generated when each type of device runs in each context attribute, and use a preset probability matrix algorithm to calculate a probability matrix of context attributes corresponding to each type of device to extract features of each type of device.

Specifically, during the normal operation of the device, the data collector can collect the IoT data generated during operation in each context attribute. The preset probability matrix algorithm is the probability detector algorithm based on the sliding window, and the probability matrix of the context attributes corresponding to each device, that is, each type of device in each situation (each situation corresponds to a behavior, that is, corresponds to a The probability matrix under the context attribute). The probability matrix is the feature matrix, which is used to extract the features of each type of equipment.

Step S3: According to all context attributes of all types of devices, obtain the contexts shared by all devices in the neighborhood range, and use them as neighborhood shared contexts.

Specifically, all context attributes of all types of devices can be merged through reduction, and the context attributes shared by all devices in the neighborhood range can be obtained, which is called neighborhood shared context.

Step S4: According to the context shared by all devices in the neighborhood range, a neighborhood-device context correspondence table is formed, wherein the correspondence table includes a probability matrix.

Specifically, the rows of the neighborhood-device context correspondence table represent each neighborhood shared context, the columns represent each type of device, and the content in the correspondence table is a probability matrix. That is to say, each neighborhood context corresponds to which context of each type of device, and the correspondence table stores the probability matrix of this context of this device.

Steps S1 to S4 of the present invention are the construction process of the context-aware anomaly detection model. Refer to FIG. 2 for details, which shows the specific implementation process, including: device type classification, definition of context attributes, calculation of probability matrix according to the context of the device, A neighborhood shared context attribute is formed, and a neighborhood shared context-equipment behavior probability matrix correspondence table (neighborhood-equipment context correspondence table) is formed.

Step S5: Calculate the possibility that the neighborhood where the device to be detected is located is in each neighborhood shared context, and determine the target neighborhood shared context corresponding to the neighborhood based on the calculated possibility.

Specifically, in the actual detection process, all devices will continuously send data packets to the anomaly detection center, and devices in different neighborhoods will be detected separately.

In the specific implementation process, the data packet sent by the IoT device to the anomaly detection center should contain three contents: device type, neighborhood, and IoT data, that is, DataPackage=(DeviceType, Area, Data). DeviceType represents the type of device, and the anomaly detection center needs to load different behavioral feature models (probability matrices) for different devices for detection; Area represents the neighborhood, the same neighborhood shares the same context, so the anomaly detection center needs to load the same neighborhood. The data is comprehensively analyzed, and different neighborhoods are independent of each other; Data represents the data to be detected, which is the focus of the present invention, and the sequence of data generated in a short time is sent to the anomaly detection center to adapt to the sliding window detection method .

Specifically, the aforementioned preset probability matrix algorithm can be used to calculate the possibility that the current situation belongs to each neighborhood shared context, so as to determine the target neighborhood shared context.

Step S6: According to the determined target neighborhood shared context, load a probability matrix corresponding to the device to be detected from the neighborhood-device context correspondence table.

Specifically, the probability matrix loaded in this step is the probability matrix corresponding to the context attribute of each type of device in the current environment determined in step S5.

Step S7: Based on the loaded probability matrix, a preset probability matrix algorithm is used to perform anomaly detection on the data of each device to be detected.

Specifically, anomaly detection is performed by using the corresponding probability matrix for the environment where different devices are located. The main contribution of the present invention is that it utilizes the feature that the neighboring devices in the Internet of Things system share the same context attribute, and can comprehensively analyze and determine the shared context attribute of all the devices in the same neighborhood, which improves the credibility of the context judgment process, that is, improves the accuracy.

In one embodiment, step S2 specifically includes:

Separate and independently train different contexts of different types of devices;

The normal operation data of all situations is collected by the data collector, and divided into no more than 10 segments according to its value range, wherein the symbol representing the segment to which the data belongs is used to represent a data;

The collected data is formed into a sequence with time as the dimension, and converted into a sequence of symbols, and a sliding window W with a fixed size n is defined to make it move in the direction of time flow;

At each moment, there is a character sequence of length n in the sliding window, count the number of double-character pairs with distances from 1 to n-1, and create a kind of double-character behavior with adjacent distances from 1 to 1. n-1 is the feature matrix of the column, and the number of statistics of the sliding window during the moving process is recorded in the feature matrix;

The matrix obtained by the above calculation method is normalized by column, and the probability of each character pair appearing under this distance is obtained, which is used as the corresponding probability matrix.

Specifically, at each moment, there is a character sequence of length n in the sliding window, and count the number of double character pairs with distances from 1 to n-1, namely W[1]W[2], W[2 ]W[3],...,W[n-1]W[n],...,W[1]W[3],W[2]W[4],...,W[n-1]W[n ]. Among them, the normalization process is that each record is divided by the sum of the columns to obtain the probability of each character pair appearing at this distance, that is, the probability matrix S (behavioral characteristics) of such a device in this context, and S is used for subsequent contextual determination and anomaly detection.

In one embodiment, step S3 specifically includes:

Comprehensively analyze all the context attributes, directly combine the unrelated contexts, reduce the repeated parts of the related contexts and then combine them to form a neighborhood shared context that includes all the contexts of all devices.

Specifically, on the basis of anomaly detection algorithm, it is necessary to determine the context attribute first. According to the characteristics of the work and deployment of the Internet of Things system, all devices in the same neighborhood share the same context attribute, and the present invention determines the neighborhood shared context attribute according to this feature, and the specific implementation process is as follows:

其中每一行表示邻域共享上下文属性的种类，每一行中的内容表示该邻域共享上下文对应于各类设备上下文的特征，即S_A1表示设备A在上下文1中的概率矩阵。Predefine the context attributes of all types of devices, comprehensively analyze all context attributes, directly combine unrelated contexts, and reduce related contexts to their repeated parts and then combine them to form a neighborhood shared context that includes all contexts of all devices. The analysis results form a neighborhood-device context correspondence table, which can be expressed as a matrix, for example

Each row represents the type of neighborhood shared context attributes, and the content in each row represents the characteristics of the neighborhood shared context corresponding to various device contexts, that is, S _A1 represents the probability matrix of device A in context 1.

In one embodiment, step S5 specifically includes:

According to the neighborhood-device context correspondence table, a preset probability matrix algorithm is used to calculate the possibility that the neighborhood where the device to be detected is located is in the shared context of each neighborhood;

And take the most likely context as the target neighborhood to share the context.

Among them, the calculation method of the possibility that the neighborhood where the device to be detected is located is in each neighborhood shared context includes:

min{P(i)}

P(i)=a*D(M _A ,S _Ai )+b*D(M _B ,S _Bi )+c*D(M _C ,S _Ci )+…

Among them, i represents the neighborhood sharing context number, a, b, c represent the number of devices A, B, and C in the neighborhood, D represents the function to calculate the Euclidean distance between two matrices, and S _Ai represents the neighborhood sharing context The probability matrix of device A corresponding to attribute i, S _Bi the probability matrix of device B corresponding to the neighborhood shared context attribute i, S _Ci represents the probability matrix of device C corresponding to the neighborhood shared context attribute i, M _A represents all A-type devices The average value of the probability matrix of , M _B represents the average value of the probability matrix of all B-type devices, and M _C represents the average value of the probability matrix of all C-type devices.

Specifically, when the neighborhood context detection period is reached, the probability matrix of the test data is extracted from all the data within a period of time before the time using the same method as the feature matrix extracted by the probability matrix algorithm, and then the same The probability matrices extracted by the kind of devices are averaged, for example, the average matrix M _A of device A, and the average matrix M _B of device B. According to the neighborhood-device context correspondence table obtained by the previous analysis, the possibility of belonging to the shared context of each neighborhood is calculated. For example, if the neighborhood sharing context 1 corresponds to S _A1 , S _B1 , and S _C1 , the possibility calculation formula is: P(1)=a*D(M _A ,S _A1 )+b*D(M _B ,S _B1 )+c*D( _{MC , S C1 )} , where a, b, and c represent the number of devices A, B, and C in the neighborhood, and D is a function to calculate the Euclidean distance between two matrices. For each row in the neighborhood-device context correspondence table, the corresponding device context probability matrix is extracted to calculate the P value, and its size is compared. The largest value is determined as the context attribute of the neighborhood in the next cycle.

In one embodiment, step S7 specifically includes:

After the data of the device to be detected is converted into a character sequence, a sliding window W of equal size n is defined, and the probability of occurrence of the character sequence in the sliding window is calculated;

The calculated probability value is compared with the set threshold p. If it is smaller than the threshold value, it is marked as abnormal, and the window continues to slide. If k abnormal moments appear continuously, the data at this moment is detected as abnormal.

Specifically, according to the detection method of the present invention, in the process of detecting abnormal behavior of Internet of Things data, it is first necessary to determine in which context the current data of the device is generated. After the context is determined, the probability matrix S corresponding to the context of the device is selected from all the extracted high-green matrices. After converting the data to be detected into character sequences, define a sliding window W of equal size n, and calculate the occurrence probability of the sequence in the sliding window. The calculation method is to read the corresponding probability in the probability matrix and then multiply, for example: define S(AB ,n-1) represents the probability of the AB character pair appearing at the distance of n-1, then for a sliding window of size 4, where the sequence is ABAC, the probability calculation formula is: S(AB,1)*S(BA ,1)*S(AC,1)*S(AA,2)*S(BC,2)*S(AC,3). The obtained probability value is compared with the set threshold p. If it is less than the threshold value, it is marked as abnormal, and the window continues to slide. If k abnormal moments appear in a row, the data at this moment is detected as abnormal. The values of p and k need to be set in advance according to the training data, and their values are closely related to the size n of the sliding window and the characteristics of the device itself.

It can be seen from the method provided by the present invention that the improved probability matrix algorithm is adopted, and the obtained probability matrix is not only used to calculate the normal probability of data behavior, but also used as the feature of the data itself to determine context attributes and abnormal data. pair detection.

In one embodiment, when performing anomaly detection, the step of determining the shared context of the target neighborhood is performed every preset period.

By using the context environment changing speed much lower than the abnormal detection frequency, it is not necessary to determine the neighborhood shared context every time it is detected, but to take an appropriate period to determine, which can reduce the amount of detection calculation.

In order to more clearly illustrate the implementation process of the method provided by the present invention, a specific example is used for detailed description below, please refer to FIG. 3 .

First, all devices are divided into fields, and then the devices in different neighborhoods are detected separately. For a neighborhood, it is judged that the neighborhood has shared context attributes, and the corresponding probability matrix is loaded according to the corresponding table and the judged shared context attributes. , and then use the probability matrix algorithm (that is, the preset probability matrix algorithm) to detect the abnormality, and judge whether the abnormality is detected, if it is, then alarm, if not, then judge whether the data detection is over, if it is over, then end the detection process , if not, further judge whether the context detection period is reached, if so, continue to return to the step of judging shared context attributes, otherwise continue to perform data detection.

Based on the same inventive concept, the present application also provides a device corresponding to the data anomaly detection method in the Internet of Things big data environment in the first embodiment. For details, refer to the second embodiment.

Embodiment 2

This embodiment provides a data anomaly detection device in the big data environment of the Internet of Things, please refer to FIG. 4 , the device includes:

The context attribute definition module 201 is used to classify the IoT devices to be detected, and define a context attribute for each type of device, wherein each context attribute corresponds to a behavior pattern of the corresponding type of device;

The device feature extraction module 202 is used to collect the IoT data generated when each type of device runs in each context attribute, and use a preset probability matrix algorithm to calculate the probability matrix of the context attribute corresponding to each device, so as to extract each type of device. characteristics of the class of equipment;

The neighborhood sharing context obtaining module 203 is used to obtain the context shared by all devices within the neighborhood scope according to all the context attributes of all kinds of devices, and use it as the neighborhood sharing context;

A correspondence table forming module 204, configured to form a neighborhood-device context correspondence table according to the context shared by all devices in the neighborhood, wherein the correspondence table includes a probability matrix;

The target neighborhood shared context determination module 205 is used to calculate the possibility that the neighborhood where the device to be detected is located is in each neighborhood shared context, and based on the calculated possibility, determine the target neighborhood corresponding to the neighborhood shared context;

The probability matrix loading module 206 is configured to load the probability matrix corresponding to the device to be detected from the neighborhood-device context correspondence table according to the determined target neighborhood sharing context;

The abnormality detection module 207 is configured to perform abnormality detection on the data of each device to be detected by using a preset probability matrix algorithm based on the loaded probability matrix.

In one embodiment, the device feature extraction module 202 is specifically configured to:

Separate and independently train different contexts of different types of devices;

The normal operation data of all situations is collected by the data collector, and divided into no more than 10 segments according to its value range, wherein the symbol representing the segment to which the data belongs is used to represent a data;

The collected data is formed into a sequence with time as the dimension, and converted into a sequence of symbols, and a sliding window W with a fixed size n is defined to make it move in the direction of time flow;

At each moment, there is a character sequence of length n in the sliding window, count the number of double-character pairs with distances from 1 to n-1, and create a kind of double-character behavior with adjacent distances from 1 to 1. n-1 is the feature matrix of the column, and the number of statistics of the sliding window during the moving process is recorded in the feature matrix;

The matrix obtained by the above calculation method is normalized by column, and the probability of each character pair appearing under this distance is obtained, which is used as the corresponding probability matrix.

In one embodiment, the neighborhood sharing context obtaining module 203 is specifically configured to:

Comprehensively analyze all the context attributes, directly combine the unrelated contexts, reduce the repeated parts of the related contexts and then combine them to form a neighborhood shared context that includes all the contexts of all devices.

In one embodiment, the target neighborhood shared context determination module 205 is specifically configured to:

According to the neighborhood-device context correspondence table, a preset probability matrix algorithm is used to calculate the possibility that the neighborhood where the device to be detected is located is in the shared context of each neighborhood;

And take the most likely context as the target neighborhood to share the context.

In one embodiment, the possibility calculation in the target neighborhood shared context determination module 205 specifically includes:

min{P(i)}

P(i)=a*D(M _A ,S _Ai )+b*D(M _B ,S _Bi )+c*D(M _C ,S _Ci )+…

Among them, i represents the neighborhood sharing context number, a, b, c represent the number of devices A, B, and C in the neighborhood, D represents the function to calculate the Euclidean distance between two matrices, and S _Ai represents the neighborhood sharing context The probability matrix of device A corresponding to attribute i, S _Bi the probability matrix of device B corresponding to the neighborhood shared context attribute i, S _Ci represents the probability matrix of device C corresponding to the neighborhood shared context attribute i, M _A represents all A-type devices The average value of the probability matrix of , M _B represents the average value of the probability matrix of all B-type devices, and M _C represents the average value of the probability matrix of all C-type devices.

In one embodiment, the anomaly detection module 207 is specifically used to:

After the data of the device to be detected is converted into a character sequence, a sliding window W of equal size n is defined, and the probability of occurrence of the character sequence in the sliding window is calculated;

The calculated probability value is compared with the set threshold p. If it is smaller than the threshold value, it is marked as abnormal, and the window continues to slide. If k abnormal moments appear continuously, the data at this moment is detected as abnormal.

In an implementation manner, the apparatus provided in this embodiment further includes a period detection module, configured to perform the step of determining the shared context of the target neighborhood every preset period during abnormality detection.

In order to more clearly illustrate the structure of the device provided by the present invention, a specific example is used for detailed description below, please refer to FIG. 5 .

In Figure 5, training data is collected from the device group for model training through data collection. During the actual detection process, the devices in each neighborhood will send real-time IoT data to the anomaly detection center for subsequent anomaly detection.

The anomaly detection center is equivalent to the detection device in this embodiment, the data behavior training module is equivalent to the correspondence table forming module 204, and is used to construct a detection model, the neighborhood context determination module is equivalent to the target neighborhood shared context determination module 205, and the data abnormality detection module The module corresponds to the abnormality detection module 207 .

Since the apparatus introduced in the second embodiment of the present invention is an apparatus used to implement the data anomaly detection method in the IoT big data environment in the first embodiment of the present invention, based on the method described in the first embodiment of the present invention, it belongs to the field of Personnel can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method of Embodiment 1 of the present invention belong to the scope of protection of the present invention.

Embodiment 3

Based on the same inventive concept, the present application also provides a computer-readable storage medium 300, see FIG. 6, on which a computer program 311 is stored, which implements the method in Embodiment 1 when the program is executed.

Since the computer-readable storage medium introduced in the third embodiment of the present invention is the computer-readable storage medium used to implement the data anomaly detection method in the IoT big data environment in the first embodiment of the present invention, it is based on the first embodiment of the present invention. For the introduced method, those skilled in the art can understand the specific structure and modification of the computer-readable storage medium, so it is not repeated here. Any computer-readable storage medium used in the method in Embodiment 1 of the present invention falls within the scope of protection of the present invention.

Embodiment 4

Based on the same inventive concept, the present application also provides a computer device, see FIG. 7 , including a storage 401, a processor 402, and a computer program 403 stored in the memory and running on the processor, and the processor 402 executes the above program When the method in the first embodiment is implemented.

Since the computer equipment introduced in the fourth embodiment of the present invention is the computer equipment used to implement the data anomaly detection method in the IoT big data environment in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the field of Those who belong to it can understand the specific structure and deformation of the computer equipment, so it is not repeated here. All computer equipment used in the method in Embodiment 1 of the present invention belongs to the scope of protection of the present invention.

Although the preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of the present invention.

Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, provided that these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims (8)

1. A data anomaly detection method under an Internet of things big data environment is characterized by comprising the following steps:

step S1: classifying the Internet of things equipment to be detected, and defining context attributes for each type of equipment, wherein each context attribute corresponds to one behavior mode of the type of equipment;

step S2: acquiring Internet of things data generated when each type of equipment operates in each context attribute, and calculating a probability matrix of the context attribute corresponding to each type of equipment by using a preset probability matrix algorithm so as to extract the characteristics of each type of equipment;

step S3: obtaining a context shared by all the devices in the neighborhood range according to all the context attributes of all the types of devices, and taking the context as a neighborhood shared context;

step S4: forming a neighborhood-equipment context corresponding table according to the context shared by all equipment in the neighborhood range, wherein the corresponding table comprises a probability matrix;

step S5: calculating the possibility that the neighborhood where the equipment to be detected is located in each neighborhood sharing context, and determining a target neighborhood sharing context corresponding to the neighborhood based on the calculated possibility;

step S6: according to the determined target neighborhood sharing context, loading a probability matrix corresponding to the equipment to be detected from a neighborhood-equipment context corresponding table;

step S7: based on the loaded probability matrix, performing anomaly detection on the data of each device to be detected by adopting a preset probability matrix algorithm;

wherein, step S5 specifically includes:

calculating the possibility that the neighborhood where the equipment to be detected is located is in each neighborhood sharing context by adopting a preset probability matrix algorithm according to the neighborhood-equipment context corresponding table;

taking the context with the highest possibility as a target neighborhood sharing context;

the calculation method of the possibility that the neighborhood where the equipment to be detected is located shares the context in each neighborhood comprises the following steps:

min{P(i)}

P(i)＝a*D(M_A,S_Ai)+b*D(M_B,S_Bi)+c*D(M_C,S_Ci)+…

wherein i represents the serial number of the shared context in the neighborhood, a, B and C represent the number of the equipment A, the equipment B and the equipment C in the neighborhood, D represents a function for calculating the Euclidean distance between two matrixes, and S_AiProbability matrix, S, representing devices A corresponding to the neighborhood sharing context attribute i_BiProbability matrix of device B corresponding to neighborhood sharing context attribute i, S_CiProbability matrix, M, representing devices C corresponding to the neighborhood sharing context attribute i_AMean value of probability matrix representing all class A devices, M_BMean value of probability matrix representing all B-class devices, M_CRepresents the average of the probability matrices for all class C devices.

2. The method according to claim 1, wherein step S2 specifically comprises:

training different contexts of different kinds of equipment separately and independently;

collecting normal operation data of all conditions through a data collector, and dividing the normal operation data into no more than 10 segments according to the value range of the normal operation data, wherein a symbol representing the segment to which the data belongs is used for representing one data;

forming a sequence taking time as a dimension on the acquired data, converting the sequence into a symbol sequence, and defining a sliding window W with a fixed size of n to move along the time flow direction;

at each moment, a character sequence with the length of n exists in the sliding window, the number of double character pairs with the distance of 1 to n-1 is counted, a feature matrix with the type of the double character pairs as rows and the adjacent distance of 1 to n-1 as columns is created, and the counted number of the sliding window in the moving process is recorded in the feature matrix;

and normalizing the matrix obtained by the calculation mode according to columns to obtain the probability of each character pair at the distance, and taking the probability as a corresponding probability matrix.

3. The method according to claim 2, wherein step S3 specifically comprises:

comprehensively analyzing all context attributes, directly combining the unrelated contexts, reducing the repeated parts of the related contexts and then combining the related contexts to form a neighborhood sharing context containing all the contexts of all the equipment.

4. The method according to claim 1, wherein step S7 specifically comprises:

after converting the data of the equipment to be detected into a character sequence, defining sliding windows W with equal size n, and calculating the probability of the character sequence in the sliding windows;

and comparing the calculated probability value with a set threshold value p, marking the abnormal time if the probability value is less than the threshold value, continuing sliding the window, and detecting that the data at the time is abnormal if k abnormal times continuously appear.

5. The method of claim 1, wherein the step of determining the target neighborhood sharing context is performed every predetermined period when performing anomaly detection.

6. The utility model provides a data anomaly detection device under thing networking big data environment which characterized in that includes:

the context attribute definition module is used for classifying the Internet of things equipment to be detected and defining context attributes for each type of equipment, wherein each context attribute corresponds to one behavior mode of the type of equipment;

the device feature extraction module is used for acquiring Internet of things data generated when each type of device operates in each context attribute, calculating a probability matrix of the context attribute corresponding to each type of device by using a preset probability matrix algorithm, and extracting features of each type of device;

the neighborhood sharing context obtaining module is used for obtaining the context shared by all the devices in the neighborhood range according to all the context attributes of all the types of devices, and taking the context as the neighborhood sharing context;

a correspondence table forming module, configured to form a neighborhood-device context correspondence table according to a context shared by all devices in a neighborhood range, where the correspondence table includes a probability matrix;

the target neighborhood sharing context determining module is used for calculating the possibility that the neighborhood where the equipment to be detected is located in each neighborhood sharing context, and determining the target neighborhood sharing context corresponding to the neighborhood based on the calculated possibility;

the probability matrix loading module is used for loading a probability matrix corresponding to the equipment to be detected from the neighborhood-equipment context corresponding table according to the determined target neighborhood sharing context;

the anomaly detection module is used for carrying out anomaly detection on the data of each device to be detected by adopting a preset probability matrix algorithm based on the loaded probability matrix;

the target neighborhood sharing context determining module is specifically configured to:

calculating the possibility that the neighborhood where the equipment to be detected is located is in each neighborhood sharing context by adopting a preset probability matrix algorithm according to the neighborhood-equipment context corresponding table;

taking the context with the highest possibility as a target neighborhood sharing context;

the calculation method of the possibility that the neighborhood where the equipment to be detected is located shares the context in each neighborhood comprises the following steps:

min{P(i)}

P(i)＝a*D(M_A,S_Ai)+b*D(M_B,S_Bi)+c*D(M_C,S_Ci)+…

wherein i represents the serial number of the shared context in the neighborhood, a, B and C represent the number of the devices A, B and C in the neighborhood, and D represents the calculation of the Europe between the two matrixesFunction of formula distance, S_AiProbability matrix, S, representing devices A corresponding to the neighborhood sharing context attribute i_BiProbability matrix of device B corresponding to neighborhood sharing context attribute i, S_CiProbability matrix, M, representing devices C corresponding to the neighborhood sharing context attribute i_AMean value of probability matrix representing all class A devices, M_BMean value of probability matrix representing all B-class devices, M_CRepresents the average of the probability matrices for all class C devices.

7. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when executed, implements the method of any one of claims 1 to 5.

8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 5 when executing the program.

Images