CN112671791A - Method, device, computer equipment and readable storage medium for detecting entity abnormity - Google Patents
Method, device, computer equipment and readable storage medium for detecting entity abnormity Download PDFInfo
- Publication number
- CN112671791A CN112671791A CN202011603971.7A CN202011603971A CN112671791A CN 112671791 A CN112671791 A CN 112671791A CN 202011603971 A CN202011603971 A CN 202011603971A CN 112671791 A CN112671791 A CN 112671791A
- Authority
- CN
- China
- Prior art keywords
- time length
- unit time
- entity
- characteristic
- score
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention provides a method, a device, computer equipment and a readable storage medium for detecting entity abnormity. The method comprises the following steps: acquiring a network flow of an entity to be detected; extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample; traversing a preset bipartite tree structure by using the characteristic sample, and determining a terminal node corresponding to the characteristic sample in the bipartite tree structure to obtain a target node; determining the abnormal score of the characteristic sample according to the distribution of the target node and the corresponding terminal nodes of the plurality of historical characteristic samples in the binary tree structure; and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample. The invention can realize the abnormity detection of the real-time network data flow.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and an apparatus for detecting entity anomalies, a computer device, and a readable storage medium.
Background
With the comprehensive arrival of the big data era, the network environment is diversified, the information amount of user personnel is increased rapidly, the business of enterprises is continuously expanded, and the boundary of network security is fuzzy and gradually disappears. Enterprise internal data, which is of paramount importance among enterprise assets, also faces ever changing and evolving security threats. A User and Entity Behavior Analysis (UEBA) system is used as a comprehensive abnormal User monitoring system which is continuously improved at present, and aims to break through the traditional safety detection scheme.
The traditional safety detection method based on rules, characteristic signatures, manual analysis and the like in the traditional safety detection scheme has the following problems: the method can not automatically adapt to the continuously changed attack mode of the intruder, thereby failing to provide the safety alarm in real time. In order to solve the problem, the prior art proposes an anomaly detection method based on algorithms such as isolated forests, support vector machines, local anomaly factors and the like, but the methods are all suitable for batch data and are in an online-to-offline mode, and although the method has an excellent anomaly identification principle, the method is lack of capturing flow data characteristics and cannot meet the anomaly detection of online real-time data streams.
Therefore, how to implement the anomaly detection of the real-time network data flow becomes a technical problem to be solved urgently in the field.
Disclosure of Invention
The invention aims to provide a method, a device, a computer device and a readable storage medium for detecting entity abnormity, which are used for solving the technical problems in the prior art.
In one aspect, the present invention provides a method for detecting entity anomalies.
The method for detecting entity abnormality comprises the following steps: acquiring a network flow of an entity to be detected; extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample; traversing a preset bipartite tree structure by using a characteristic sample, and determining a terminal node corresponding to the characteristic sample in the bipartite tree structure to obtain a target node; determining the abnormal score of the characteristic sample according to the distribution of the target node and a plurality of historical characteristic samples in the corresponding terminal nodes in the binary tree structure, wherein the historical characteristic samples are positioned in front of the characteristic sample; and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample.
Further, the step of determining the abnormal score of the characteristic sample according to the distribution of the target node and the corresponding terminal nodes of the plurality of historical characteristic samples in the binary tree structure includes: determining the depth of the target node; searching for the density at the target node in the density distribution of the plurality of historical feature samples recorded by the bipartite tree structure, wherein the density distribution comprises the density at each terminal node in the bipartite tree structure, and the density is the number of feature samples distributed at the terminal node by the plurality of historical feature samples; and calculating the abnormal score of the characteristic sample according to the depth and the density of the target node.
Further, the abnormality score S of the feature sample is calculated using the following formula:
S＝Den*2Dep
and Den is the depth of the target node, and Dep is the corresponding density of the target node.
Further, presetting a reference window comprising N characteristic samples; the plurality of historical feature samples are located in the reference window; after the step of determining an abnormality score for the sample of features, the method of detecting entity abnormalities further comprises: storing the characteristic sample into a cache window, and acquiring an abnormal score corresponding to the next characteristic sample; and when the number of the characteristic samples in the cache window reaches N, replacing the characteristic samples in the reference window with the characteristic samples in the cache window, and emptying the cache window.
Further, presetting a plurality of binary tree structures, and respectively obtaining the abnormal score of the characteristic sample corresponding to each binary tree structure; according to the abnormal score of the characteristic sample, the step of determining the abnormal condition of the entity to be detected in the corresponding unit time length comprises the following steps: calculating the sum of abnormal scores of the characteristic sample corresponding to all the binary tree structures to obtain a score sum; and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score.
Further, the step of determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score comprises the following steps: and judging whether the score sum is smaller than a preset score threshold value, and determining that the entity to be detected has abnormal behaviors in the corresponding unit time length when the score sum is smaller than the score threshold value.
Further, when determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score, the specifically executed steps include: and utilizing a plurality of continuous scores and construction scores and sequences, determining a local extreme value in the scores and sequences by a time sequence analysis method, and determining that the entity to be detected has abnormal behavior in unit time length corresponding to the local extreme value.
Further, the step of extracting the traffic characteristics of the network flow in unit time length to obtain a characteristic sample includes: calculating the number of the received timestamps in the unit time length; calculating the flow of the network flow in the unit time length; and constructing the traffic characteristics of the network flow in the unit time length according to the quantity and the traffic.
Further, the step of extracting the traffic characteristics of the network flow in unit time length to obtain a characteristic sample further includes: dividing network flows in a preset time length into a plurality of network flows in the unit time length, wherein the preset time length comprises the unit time lengths; the step of constructing the traffic characteristics of the network flow in the unit time length according to the number and the traffic comprises the following steps: normalizing the number according to the maximum number and the minimum number corresponding to each unit time length in the preset time length; normalizing the flow according to the maximum flow and the minimum flow corresponding to each unit time length in the preset time length; and taking the normalized quantity and the normalized flow as the flow characteristics of the network flow in the unit time length.
In another aspect, the present invention provides an apparatus for detecting entity abnormality.
The apparatus for detecting entity abnormality includes: the acquisition module is used for acquiring the network flow of the entity to be detected; the extraction module is used for extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample; the first processing module is used for traversing a preset bipartite tree structure by using a characteristic sample and determining a corresponding terminal node of the characteristic sample in the bipartite tree structure to obtain a target node; the second processing module is used for determining the abnormal score of the characteristic sample according to the distribution of the target node and a plurality of historical characteristic samples in the corresponding terminal nodes in the binary tree structure, wherein the historical characteristic samples are positioned in front of the characteristic sample; and the third processing module is used for determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample.
In another aspect, to achieve the above object, the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, the steps of the method are implemented.
In a further aspect, to achieve the above object, the present invention further provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the above method.
The method, the device, the computer equipment and the readable storage medium for detecting the entity abnormity, provided by the invention, are used for acquiring the network flow of the entity to be detected, extracting the flow characteristics of the network flow in unit time length to obtain characteristic samples, traversing each characteristic sample to be detected through a preset bipartite tree structure, determining a corresponding terminal node in the bipartite tree structure as a target node, determining the abnormity score of the characteristic sample according to the distribution of the target node and the corresponding terminal nodes of a plurality of historical characteristic samples in the bipartite tree structure, and finally determining the abnormity condition of the entity to be detected in the corresponding unit time length according to the abnormity score of the characteristic sample. By the method and the device, the link flow abnormity detection in the user entity behavior analysis is realized based on the bipartite tree structure, and the abnormity detection of the real-time network data flow is also realized.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a block diagram illustrating a method for detecting entity anomalies according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a binary tree structure according to an embodiment of the present invention;
FIGS. 3 and 4 are schematic diagrams of the construction of the bipartite tree structure shown in FIG. 2;
fig. 5 and fig. 6 are schematic distribution diagrams of corresponding terminal nodes in a binary tree structure of a plurality of history feature samples according to an embodiment of the present invention;
FIG. 7 is a block diagram of an apparatus for detecting entity anomalies according to a second embodiment of the present invention;
fig. 8 is a hardware configuration diagram of a computer device according to a third embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In order to realize the abnormal detection of the entity behavior based on the real-time network data flow, the application provides a method, a device, computer equipment and a readable storage medium for detecting the entity abnormality, wherein the network flow of the entity to be detected is obtained, the flow characteristic of the network flow in unit time length is extracted and used as a characteristic sample, when one characteristic sample is detected, the characteristic sample is used for traversing a preset bipartite tree structure until reaching a terminal node of the bipartite tree structure, namely the terminal node corresponding to the characteristic sample in the bipartite tree structure, is used as a target node, then the abnormal score of the characteristic sample is determined according to the target node and the distribution of the terminal nodes corresponding to a plurality of historical characteristic samples in the bipartite tree structure, namely, the position of the characteristic sample in the bipartite tree structure is compared with the distribution of the plurality of historical characteristic samples in the bipartite tree structure, therefore, the abnormal score of the characteristic sample is determined, the abnormal condition of the entity to be detected in the unit time length corresponding to the characteristic sample is finally determined according to the abnormal score of the characteristic sample, and the change of the network flow data distribution in the short time of the unit time length can be detected. Meanwhile, as the network flow reaches the detection link, the detection is continued for a period of time, the feature samples are continuously extracted and the abnormal score is detected, so that the abnormal condition of the entity to be detected in each unit time length is obtained, and the user behavior data distribution in a long time can be captured. In summary, the application provides a method, an apparatus, a computer device and a readable storage medium for detecting entity anomalies, which can realize anomaly detection of real-time network data streams.
Specific embodiments of a method, an apparatus, a computer device and a readable storage medium for detecting entity anomalies provided by the present application will be described in detail below.
Example one
An embodiment of the present invention provides a method for detecting an entity anomaly, through which an anomaly detection on a real-time network data stream can be implemented, and specifically, fig. 1 is a method for detecting an entity anomaly provided in an embodiment of the present invention, as shown in fig. 1, the method for detecting an entity anomaly provided in the embodiment includes steps S101 to S105 as follows.
Step S101: and acquiring the network flow of the entity to be detected.
Specifically, a detection link may be set to obtain a network flow uploaded and/or downloaded by the entity to be detected, so as to determine whether the entity to be detected has an abnormal behavior through the network flow.
Step S102: and extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample.
Specifically, a certain time length is set as a unit time length, the unit time length may be 30s or 60s, for example, and different time lengths may be specifically selected according to different actual service scenarios. In the step, the network traffic in a unit time length is intercepted in real time, the traffic characteristics of the network traffic are extracted, the traffic characteristics comprise the size of the traffic, the speed of the traffic, the number of traffic packets and the like, and the extracted traffic characteristics are defined as characteristic samples.
The network flow characteristic sampling method comprises the steps of extracting a network flow with unit time length each time, then extracting flow characteristic to form a characteristic sample, or extracting flow characteristics of the network flow in each network unit time length to form a plurality of characteristic samples after extracting network flows with a plurality of unit time lengths, and specifically determining according to real-time monitoring practical requirements and data processing capacity.
Step S103: and traversing the preset bipartite tree structure by using the characteristic sample, and determining the corresponding terminal node of the characteristic sample in the bipartite tree structure to obtain the target node.
And presetting a bipartite tree structure, wherein each characteristic sample can reach one terminal node of the bipartite tree structure by traversing the bipartite tree structure. Specifically, when constructing the bipartite tree structure, a working space of the bipartite tree structure is initialized, a maximum value array and a minimum value array of a working range of the bipartite tree structure and a depth of the tree are determined, then a dimension is randomly selected in the working space to generate an internal node, the node is used as a splitting point, and a characteristic (maximum value + minimum value)/2 of the dimension is selected to establish the bipartite tree structure.
In step S103, for the feature sample to be processed currently, the constructed bipartite tree structure is traversed by using the feature sample until reaching the terminal node of the bipartite tree structure, the traversal is completed, and the terminal node reached by the feature sample is determined to be the target node.
Fig. 2 is a schematic diagram of a treelet structure according to an embodiment of the present invention, and fig. 3 and 4 are schematic diagrams of a structure for constructing the treelet structure shown in fig. 2, and as shown in fig. 2, a treelet structure is constructed in a two-dimensional space, where the tree has a depth of 2, and includes 7 nodes of A, B1, B2, and C1-C4, where C1-C4 are terminal nodes. As shown in FIG. 3, the x-y two-dimensional space is divided into two parts of x <0.5 and x ≧ 0.5 by the node A, as shown in FIG. 4, the part of x <0.5 is divided into two parts of x <0.5 and y <0.5 and x <0.5 and y ≧ 0.5 by the node B1, and the part of x ≧ 0.5 is divided into two parts of 0.75> x ≧ 0.5 and x ≧ 0.75 by the node B2.
For example, for one feature sample (0.8,0.3), when traversing the binary tree structure, starting from the root node a, the node a reaches the node B2 and then reaches the node C4, and the node C4 is also the terminal node reached by the feature sample (0.8,0.3), i.e. the destination node of the feature sample (0.8, 0.3).
For another example, for one feature sample (0.15,0.9), when traversing the binary tree structure, starting from the root node a, the node a reaches the node B1 and then reaches the node C2, and the node C2 is also the terminal node reached by the feature sample (0.15,0.9), i.e., the destination node of the feature sample (0.15, 0.9).
Step S104: and determining the abnormal score of the characteristic sample according to the distribution of the target node and the corresponding terminal nodes of the plurality of historical characteristic samples in the binary tree structure.
Specifically, according to the time attribute of the network flow, the network flow time corresponding to the historical feature samples is located before the network flow time corresponding to the feature samples, and may be multiple feature samples adjacent to the feature samples, or may be a plurality of feature samples spaced from the feature samples, and each historical feature sample may be adjacent to or non-adjacent to each other; or, according to the sending and receiving position attribute of the data packet in the network flow, the data packet corresponding to the historical characteristic sample is positioned in front of the data packet corresponding to the characteristic sample.
And respectively traversing a plurality of historical characteristic samples before the characteristic sample through the binary tree structure to obtain the terminal node of each historical characteristic sample in the binary tree structure, thereby forming the distribution of the corresponding terminal nodes of the plurality of historical characteristic samples in the binary tree structure and obtaining the quantity of the historical characteristic samples distributed in different terminal nodes of the binary tree structure. Based on the fact that the abnormal behavior of the entity is less relative to the normal behavior, and correspondingly, the network flow of the abnormal behavior is less relative to the network flow of the normal behavior, that is, when the number of the distributed characteristic samples in a certain terminal node is more, the probability of the abnormal behavior of the entity to be detected in the unit time length corresponding to the characteristic sample in the terminal node is lower, when the number of the distributed characteristic samples in a certain terminal node is less, the probability of the abnormal behavior of the entity to be detected in the unit time length corresponding to the characteristic sample in the terminal node is higher, based on this, if the number of the distributed historical characteristic samples in the terminal node corresponding to the characteristic sample on the binary tree structure, that is, the number of the distributed historical characteristic samples in the target node is more, the probability of the abnormal behavior of the entity to be detected in the unit time length corresponding to the characteristic sample is lower, the probability of the abnormal behavior of the entity to be detected in the unit time length corresponding to the characteristic sample is higher, therefore, the abnormal score of the characteristic sample can be calculated by two factors of the distribution of the target node and the corresponding terminal nodes of the plurality of historical characteristic samples in the binary tree structure. For example, the ratio of the number of historical feature samples distributed in the target node to a plurality of historical feature samples may be calculated as the abnormal score of the feature samples; for another example, the anomaly score and the like may be calculated according to the depth of the target node and the number of historical feature samples distributed in the target node.
Fig. 5 and 6 are schematic distribution diagrams of terminal nodes corresponding to a plurality of historical feature samples in a binary tree structure according to an embodiment of the present invention, and as shown in fig. 5 and 6, 2 historical feature samples are distributed in a C1 node, 5 historical feature samples are distributed in a C2 node, 3 historical feature samples are distributed in a C3 node, and 8 historical feature samples are distributed in a C4 node, if a feature sample is (0.2,0.4), a target node thereof is a C1 node, and if the number of historical feature samples distributed in the target node is a ratio of the plurality of historical feature samples as an anomaly score of the feature sample, an anomaly score of the feature sample (0.2,0.4) is calculated as 2/18.
Step S105: and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample.
Specifically, in this step, the probability that the entity to be detected is abnormal in the corresponding unit duration may be determined according to the abnormality score of the feature sample, where if the lower the abnormality score of the feature sample is, the higher the probability that the entity to be detected is abnormal in the corresponding unit duration is, the higher the abnormality score of the feature sample is, and the lower the probability that the entity to be detected is abnormal in the corresponding unit duration is, the multiple abnormality scores obtained within a period of time may be sorted, and it is determined that the higher the probability that the entity to be detected is abnormal in the unit duration corresponding to the lower N previous abnormality scores is. Or, whether the entity to be detected in the corresponding unit time length is abnormal or not can be judged according to the abnormal score of the characteristic sample and a preset score threshold, wherein when the abnormal score of the characteristic sample is smaller than the score threshold, the entity to be detected in the corresponding unit time length is abnormal, and when the abnormal score of the characteristic sample is larger than or equal to the score threshold, the entity to be detected in the corresponding unit time length is not abnormal. Or constructing a score array by using a plurality of continuous scores, determining a local extreme value in the score array by using a time sequence analysis method, and determining that the entity to be detected has abnormal behavior in unit time length corresponding to the local extreme value.
Or, the higher the abnormality score of the characteristic sample is, the higher the probability that the entity to be detected is abnormal in the corresponding unit time length is, and the lower the abnormality score of the characteristic sample is, specifically, the relationship between the abnormality score and the probability that the entity to be detected is abnormal can be determined according to different abnormality score calculation methods, and based on the relationship, the abnormal condition of the entity to be detected in the corresponding unit time length can be determined.
In the method for detecting entity abnormality provided in this embodiment, a network flow of an entity to be detected is obtained, a traffic characteristic of the network flow in a unit time length is extracted, a characteristic sample is obtained, for each characteristic sample to be detected, a preset binary tree structure is traversed, a terminal node corresponding to the characteristic sample in the binary tree structure is determined, the terminal node serves as a target node, an abnormality score of the characteristic sample is determined according to distribution of the target node and the terminal nodes corresponding to a plurality of historical characteristic samples in the binary tree structure, and finally, according to the abnormality score of the characteristic sample, an abnormality condition of the entity to be detected in the corresponding unit time length is determined. By adopting the method for detecting entity abnormality provided by the embodiment, the link flow abnormality detection in the user entity behavior analysis is realized based on the bipartite tree structure, namely, the abnormality detection of the real-time network data flow is realized.
Optionally, in an embodiment, a plurality of binary tree structures are preset, and corresponding to each binary tree structure, the abnormal score of the feature sample is obtained respectively; the step of determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample comprises the following steps: calculating the sum of abnormal scores of the characteristic sample corresponding to all the binary tree structures to obtain a score sum; and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score.
Specifically, a bipartite tree forest comprising a plurality of bipartite tree structures is set, the abnormal scores of the characteristic examples can be obtained by adopting the steps S103 and S104 corresponding to each bipartite tree structure, and when the abnormal condition of the entity to be detected in the corresponding unit time length is determined according to the abnormal scores of the characteristic examples, the abnormal scores of the characteristic examples are accumulated to obtain the score sum of the characteristic examples, so that the abnormal condition of the entity to be detected in the corresponding unit time length is determined according to the score sum.
By adopting the method for detecting entity abnormality provided by the embodiment, the abnormality of the real-time network flow data is detected based on the plurality of binary tree structures, and the detection accuracy can be improved.
Further optionally, in an embodiment, when determining, according to the scores and the abnormal conditions of the entities to be detected in the corresponding unit time length, the specifically executed steps include: and judging whether the score sum is smaller than a preset score threshold value, and determining that the entity to be detected has abnormal behaviors in the corresponding unit time length when the score sum is smaller than the score threshold value.
By adopting the method for detecting the entity abnormity provided by the embodiment, whether the entity to be detected has the abnormal behavior in the corresponding unit time length is determined based on the score, the mode is simple, and the real-time performance of online abnormity detection is facilitated.
Optionally, in another embodiment, when determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score, the specifically executed step includes: and utilizing the continuous scores and the constructed sequence, determining local extreme values in the scores and the constructed sequence by a time sequence analysis method, and determining abnormal behaviors of the entity to be detected in unit time length corresponding to the local extreme values.
Specifically, if it is known that network traffic data is distributed in multiple segments with obviously different distributions, the window size n can be set to record a plurality of scores as a score and a sequence, and then a time series analysis method, such as an exponential smoothing method, an ARIMA, and the like, can be used to capture local dips/dips and local extrema in the score and the sequence, so as to determine that the entity to be detected has abnormal behavior within a unit time length corresponding to each feature sample.
By adopting the method for detecting entity abnormality provided by the embodiment, the accuracy of detecting the abnormal behavior of the entity to be detected can be improved for the service scene that the network flow data is distributed into a plurality of sections with different obvious distributions.
Optionally, in an embodiment, the step of determining the abnormal score of the feature sample according to the distribution of the target node and the corresponding terminal nodes of the plurality of historical feature samples in the binary tree structure includes: determining the depth of a target node; searching the density of the target node in the density distribution of the plurality of historical characteristic samples recorded by the binary tree structure, wherein the density distribution comprises the density of each terminal node in the binary tree structure, and the density is the number of the characteristic samples distributed at the terminal nodes by the plurality of historical characteristic samples; and calculating the abnormal score of the characteristic sample according to the depth and the density of the target node.
Specifically, after the target node is determined, the depth of the target node may be determined based on the position of the target node in the bipartite tree structure, as shown in fig. 5, if the target node is a C2 node, the depth of the target node is 2, and the density distribution of the multiple history feature samples recorded by the bipartite tree structure is specifically: the number of feature samples distributed by 18 historical feature samples at the node C1-C4 is 2, 5, 3 and 8, respectively, in this embodiment, for the case that the target node is the C2 node, calculating the abnormal score of the characteristic sample according to the depth 2 and the number of the characteristic samples distributed at the C1-C4 node, wherein, the smaller the depth of the target node is, the greater the possibility that the entity to be detected is abnormal in the unit time length corresponding to the characteristic sample is, the smaller the number of the characteristic samples distributed at the target node is, the greater the possibility that the entity to be detected is abnormal in the unit time length corresponding to the characteristic sample is, the positive correlation between the depth of the target node and the number of the characteristic samples distributed at the target node and the abnormal score can be specifically set, the smaller the abnormality score is, the higher the possibility that the entity to be detected is abnormal in the unit time length corresponding to the characteristic sample is; or, a negative correlation relationship between the depth of the target node and the number of the feature samples distributed at the target node and the abnormality score may be set, and the larger the abnormality score is, the higher the possibility that the entity to be detected is abnormal in the unit time length corresponding to the feature sample is.
By adopting the method for detecting entity abnormity provided by the embodiment, the entity abnormal behavior detection in the streaming data scene is faced, the online abnormity detection can be carried out on a large amount of even infinite network data streams, the complexity is low, and the online abnormity detection real-time performance is favorably improved.
Further optionally, in one embodiment, the anomaly score S for the feature sample is calculated using the following formula:
S＝Den*2Dep
where Den is the depth of the target node and Dep is the corresponding density of the target node.
By adopting the method for detecting entity abnormality provided by the embodiment, only the time complexity and the space complexity which are not more than o (1) at most are consumed when online abnormality detection is carried out on the network data stream, the calculation speed is high, and the real-time performance of online abnormality detection is favorably improved.
Optionally, in an embodiment, a reference window including N feature samples is preset; a plurality of historical feature samples are located in a reference window; after the step of determining an abnormality score for the feature sample, the method of detecting an entity abnormality further comprises: storing the characteristic sample into a cache window, and acquiring an abnormal score corresponding to the next characteristic sample; and when the number of the characteristic samples in the cache window reaches N, replacing the characteristic samples in the reference window with the characteristic samples in the cache window, and emptying the cache window.
Specifically, when a plurality of historical feature samples serving as a reference are used as a reference window, and the abnormal score of the feature sample is determined according to the distribution of the target node and the corresponding terminal nodes of the plurality of historical feature samples in the binary tree structure, that is, the abnormal score of the feature sample is determined according to the distribution of the target node and the corresponding terminal nodes of the respective feature samples in the reference window in the binary tree structure. Meanwhile, after the abnormal score of each characteristic sample is determined, the abnormal score is stored in a cache window, the next characteristic sample is obtained, determining abnormal score according to the distribution of the corresponding target node and the corresponding terminal node of each characteristic sample in the reference window in the binary tree structure, taking this as an example, as the network stream is processed in real time, the number of feature samples in the cache window gradually increases, when the number of the characteristic samples in the cache window reaches N, namely reaches the number of the characteristic samples in the reference window, the characteristic samples in the cache window are used for replacing the characteristic samples in the reference window, and the cache window is emptied, the obtained characteristic sample can determine the abnormal score according to the new historical characteristic sample in the reference window, and writing the characteristic sample into the cache window again, and so on, so as to realize the real-time processing of the network flow.
The method for detecting entity abnormity provided by the embodiment has good robustness, and can realize self-adaptive seamless detection without manually adjusting the model through the sliding of the reference window when the potential data distribution changes in the actual scene.
Optionally, in an embodiment, the step of extracting traffic characteristics of the network flow in a unit time length to obtain a characteristic sample includes: calculating the number of the received timestamps in unit time length; calculating the flow of the network flow in unit time length; and constructing the flow characteristics of the network flow in unit time length according to the quantity and the flow.
Specifically, the network flow mainly contains the following two parameters: the number of the timestamps received in unit time length, namely the number of the data packets in the unit time length, is calculated based on the two parameter characteristics when the network flow reaches the timestamp of the detection link and the flow of each timestamp node; calculating the flow of the network flow in unit time length, namely the size of a data packet in the unit time length; and adding the flow of each timestamp node in the unit time length to obtain the flow of the network flow in the unit time length so as to construct the flow characteristics of the network flow in the unit time length according to the quantity and the flow.
By adopting the method for detecting entity abnormality provided by the embodiment, the flow characteristics are constructed from two angles of the number and the size of the response data packets, so that the flow characteristics can accurately express the characteristics of the network flow, and the detection accuracy can be improved when abnormality detection is carried out based on the flow characteristics.
Optionally, in an embodiment, the step of extracting traffic characteristics of the network flow in a unit time length to obtain the characteristic sample further includes: dividing the network flow in the preset time length into the network flows in a plurality of unit time lengths, wherein the preset time length comprises a plurality of unit time lengths; the step of constructing the flow characteristics of the network flow in unit time length according to the quantity and the flow comprises the following steps: normalizing the number according to the maximum number and the minimum number corresponding to each unit time length in the preset time length; normalizing the flow according to the maximum flow and the minimum flow corresponding to each unit time length in the preset time length; and taking the normalized quantity and the normalized flow as the flow characteristics of the network flow in unit time length.
Specifically, when the flow characteristics of the network flow in the unit time length are constructed according to the quantity and the flow, the quantity normalization is realized through the maximum value and the minimum value of the quantity corresponding to each unit time length in the preset time length, and the flow normalization is realized through the maximum value and the minimum value of the flow corresponding to each unit time length in the preset time length. Wherein, the quantity and the flow can adopt the formula X ═ (X-X)min)/(Xmax-Xmin) Carrying out a normalization process of X′Is the normalized quantity (or flow), X is the quantity (or flow) before normalization, XmaxAnd XminRespectively of a predetermined number (or flow) of timeA maximum and a minimum value. Respectively mapping the number of the timestamps received in the unit time length and the flow of the network flow in the unit time length to [0-1 ] through normalization processing]And finally, constructing the flow characteristics of the network flow in unit time length by using the normalized quantity and the normalized flow. Based on this, when initializing the working space of the binary tree structure, the maximum value array of the working range of the binary tree structure is set to be s + 2. max (s,1-s), the minimum value array is set to be s + 2. max (s,1-s), and s is [0,1 []Random real numbers in (1).
By adopting the method for detecting entity abnormality provided by the embodiment, the complexity of the binary tree structure calculation can be reduced by carrying out normalization processing on the number of the received timestamps in unit time and the flow of the network flow in unit time.
Example two
Corresponding to the first embodiment, the second embodiment of the present invention provides a device for detecting entity abnormality, and accordingly, reference may be made to the first embodiment for details of technical features and corresponding technical effects, which are not described in detail in this embodiment. Fig. 7 is a block diagram of an apparatus for detecting entity abnormality according to a second embodiment of the present invention, as shown in fig. 7, the apparatus includes: the device comprises an acquisition module 201, an extraction module 202, a first processing module 203, a second processing module 204 and a third processing module 205.
The obtaining module 201 is configured to obtain a network flow of an entity to be detected; the extraction module 202 is configured to extract traffic characteristics of the network flow in unit duration to obtain a characteristic sample; the first processing module 203 is configured to traverse a preset bipartite tree structure by using a feature sample, and determine a terminal node corresponding to the feature sample in the bipartite tree structure to obtain a target node; the second processing module 204 is configured to determine an abnormal score of the feature sample according to the distribution of the target node and the corresponding terminal nodes of the multiple historical feature samples in the bipartite tree structure, where the historical feature samples are located before the feature sample; and the third processing module 205 is configured to determine, according to the abnormality score of the feature sample, an abnormal condition of the entity to be detected in a corresponding unit duration.
Optionally, in an embodiment, the second processing module 204 includes: the first determining unit is used for determining the depth of the target node; a searching unit, configured to search for a density at the target node in a density distribution of the plurality of historical feature samples recorded by the bipartite tree structure, where the density distribution includes a density at each terminal node in the bipartite tree structure, and the density is a number of feature samples distributed at the terminal node by the plurality of historical feature samples; and the first calculating unit is used for calculating the abnormal score of the characteristic sample according to the depth and the density of the target node.
Optionally, in an embodiment, the first calculating unit calculates the abnormality score S of the feature sample by using the following formula:
S＝Den*2Dep
and Den is the depth of the target node, and Dep is the corresponding density of the target node.
Optionally, in an embodiment, a reference window including N feature samples is preset; the plurality of historical feature samples are located in the reference window; the apparatus for detecting entity abnormality further comprises: the storage module is configured to store the feature sample into a cache window after the second processing module 204 determines the abnormal score of the feature sample, and obtain an abnormal score corresponding to a next feature sample; and the fourth processing module is used for replacing the feature samples in the reference window with the feature samples in the cache window and emptying the cache window when the number of the feature samples in the cache window reaches N.
Optionally, in an embodiment, a plurality of the treelet structures are preset, and the second processing module 204 is configured to obtain an anomaly score of the feature sample corresponding to each of the treelet structures; the third processing module comprises: the second calculation unit is used for calculating the sum of abnormal scores of the characteristic samples corresponding to all the bipartite tree structures to obtain a score sum; and the second determining unit is used for determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score.
Optionally, in an embodiment, when the second determining unit determines the abnormal condition of the entity to be detected in the corresponding unit time length according to the score, the specifically executed step includes: and judging whether the score sum is smaller than a preset score threshold value, and determining that the entity to be detected has abnormal behaviors in the corresponding unit time length when the score sum is smaller than the score threshold value.
Optionally, in an embodiment, when the second determining unit determines the abnormal condition of the entity to be detected in the corresponding unit time length according to the score, the specifically executed step includes: and utilizing a plurality of continuous scores and construction scores and sequences, determining a local extreme value in the scores and sequences by a time sequence analysis method, and determining that the entity to be detected has abnormal behavior in unit time length corresponding to the local extreme value.
Optionally, in an embodiment, the extraction module 202 includes: the third calculating unit is used for calculating the number of the timestamps received in the unit time length; a fourth calculating unit, configured to calculate a flow rate of the network flow in the unit time length; and the construction unit is used for constructing the traffic characteristics of the network flow in the unit time length according to the quantity and the traffic.
Optionally, in an embodiment, the extracting module 202 further includes: the dividing unit is used for dividing the network flow in a preset time length into a plurality of network flows in the unit time length, wherein the preset time length comprises the unit time lengths; when the constructing unit constructs the traffic characteristics of the network flow in the unit time length according to the number and the traffic, the specifically executed steps include: normalizing the number according to the maximum number and the minimum number corresponding to each unit time length in the preset time length; normalizing the flow according to the maximum flow and the minimum flow corresponding to each unit time length in the preset time length; and taking the normalized quantity and the normalized flow as the flow characteristics of the network flow in the unit time length.
EXAMPLE III
The embodiment also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers) capable of executing programs, and the like. As shown in fig. 8, the computer device 01 of the present embodiment at least includes but is not limited to: a memory 011 and a processor 012, which are communicatively connected to each other via a system bus, as shown in fig. 3. It is noted that fig. 8 only shows the computer device 01 having the component memory 011 and the processor 012, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead.
In this embodiment, the memory 011 (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the storage 011 can be an internal storage unit of the computer device 01, such as a hard disk or a memory of the computer device 01. In other embodiments, the memory 011 can also be an external storage device of the computer device 01, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), etc. provided on the computer device 01. Of course, the memory 011 can also include both internal and external memory units of the computer device 01. In this embodiment, the memory 011 is generally used for storing an operating system installed in the computer apparatus 01 and various application software, such as program codes of the apparatus for detecting entity abnormality in the second embodiment. Further, the memory 011 can also be used to temporarily store various kinds of data that have been output or are to be output.
The processor 012 may be a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data Processing chip in some embodiments. The processor 012 is generally used to control the overall operation of the computer device 01. In this embodiment, the processor 012 is configured to execute a program code stored in the memory 011 or process data, for example, a method of detecting an entity abnormality.
Example four
The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, a server, an App application mall, etc., on which a computer program is stored, which when executed by a processor implements corresponding functions. The computer-readable storage medium of this embodiment is used to store an apparatus for detecting entity anomalies, and when executed by a processor, the apparatus implements the method for detecting entity anomalies of the first embodiment.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (11)
1. A method for detecting entity anomalies, comprising:
acquiring a network flow of an entity to be detected;
extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample;
traversing a preset bipartite tree structure by using a characteristic sample, and determining a terminal node corresponding to the characteristic sample in the bipartite tree structure to obtain a target node;
determining the abnormal score of the characteristic sample according to the distribution of the target node and a plurality of historical characteristic samples in the corresponding terminal nodes in the binary tree structure, wherein the historical characteristic samples are positioned in front of the characteristic sample; and
and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample.
2. The method of claim 1, wherein the step of determining the abnormality score of the characteristic sample according to the distribution of the target node and the corresponding terminal nodes of the plurality of historical characteristic samples in the bipartite tree structure comprises:
determining the depth of the target node;
searching for the density at the target node in the density distribution of the plurality of historical feature samples recorded by the bipartite tree structure, wherein the density distribution comprises the density at each terminal node in the bipartite tree structure, and the density is the number of feature samples distributed at the terminal node by the plurality of historical feature samples;
and calculating the abnormal score of the characteristic sample according to the depth and the density of the target node.
3. The method for detecting entity anomalies according to claim 2, characterized in that the anomaly score S of the characteristic sample is calculated using the following formula:
S＝Den*2Dep
and Den is the depth of the target node, and Dep is the corresponding density of the target node.
4. The method of detecting entity anomalies according to claim 1,
presetting a reference window comprising N characteristic samples;
the plurality of historical feature samples are located in the reference window;
after the step of determining an abnormality score for the sample of features, the method of detecting entity abnormalities further comprises:
storing the characteristic sample into a cache window, and acquiring an abnormal score corresponding to the next characteristic sample;
and when the number of the characteristic samples in the cache window reaches N, replacing the characteristic samples in the reference window with the characteristic samples in the cache window, and emptying the cache window.
5. The method according to claim 1, wherein a plurality of said bipartite tree structures are preset, and anomaly scores of said feature samples are obtained respectively corresponding to each of said bipartite tree structures; according to the abnormal score of the characteristic sample, the step of determining the abnormal condition of the entity to be detected in the corresponding unit time length comprises the following steps:
calculating the sum of abnormal scores of the characteristic sample corresponding to all the binary tree structures to obtain a score sum;
and determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the score.
6. The method of claim 5, wherein the step of determining the abnormality of the entity to be detected in the corresponding unit time length according to the score sum comprises:
judging whether the score sum is smaller than a preset score threshold value, and determining that the entity to be detected has abnormal behavior in corresponding unit time length when the score sum is smaller than the score threshold value; and/or
And utilizing a plurality of continuous scores and construction scores and sequences, determining a local extreme value in the scores and sequences by a time sequence analysis method, and determining that the entity to be detected has abnormal behavior in unit time length corresponding to the local extreme value.
7. The method of claim 1, wherein the step of extracting the traffic characteristics of the network flow in a unit time length to obtain the characteristic sample comprises:
calculating the number of the received timestamps in the unit time length;
calculating the flow of the network flow in the unit time length; and
and constructing the flow characteristics of the network flow in the unit time length according to the quantity and the flow.
8. The method of detecting entity anomalies according to claim 7,
the step of extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample further comprises: dividing network flows in a preset time length into a plurality of network flows in the unit time length, wherein the preset time length comprises the unit time lengths;
the step of constructing the traffic characteristics of the network flow in the unit time length according to the number and the traffic comprises the following steps: normalizing the number according to the maximum number and the minimum number corresponding to each unit time length in the preset time length; normalizing the flow according to the maximum flow and the minimum flow corresponding to each unit time length in the preset time length; and taking the normalized quantity and the normalized flow as the flow characteristics of the network flow in the unit time length.
9. An apparatus for detecting entity anomalies, comprising:
the acquisition module is used for acquiring the network flow of the entity to be detected;
the extraction module is used for extracting the flow characteristics of the network flow in unit time length to obtain a characteristic sample;
the first processing module is used for traversing a preset bipartite tree structure by using a characteristic sample and determining a corresponding terminal node of the characteristic sample in the bipartite tree structure to obtain a target node;
the second processing module is used for determining the abnormal score of the characteristic sample according to the distribution of the target node and a plurality of historical characteristic samples in the corresponding terminal nodes in the binary tree structure, wherein the historical characteristic samples are positioned in front of the characteristic sample; and
and the third processing module is used for determining the abnormal condition of the entity to be detected in the corresponding unit time length according to the abnormal score of the characteristic sample.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 8 are implemented by the processor when executing the computer program.
11. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011603971.7A CN112671791B (en) | 2020-12-30 | 2020-12-30 | Method, device, computer equipment and readable storage medium for detecting entity abnormity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011603971.7A CN112671791B (en) | 2020-12-30 | 2020-12-30 | Method, device, computer equipment and readable storage medium for detecting entity abnormity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112671791A true CN112671791A (en) | 2021-04-16 |
| CN112671791B CN112671791B (en) | 2022-07-29 |
Family
ID=75410793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011603971.7A Active CN112671791B (en) | 2020-12-30 | 2020-12-30 | Method, device, computer equipment and readable storage medium for detecting entity abnormity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112671791B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102420723A (en) * | 2011-12-14 | 2012-04-18 | 南京邮电大学 | Anomaly detection method for various kinds of intrusion |
| US20170251012A1 (en) * | 2016-02-25 | 2017-08-31 | Darktrace Limited | Cyber security |
| CN110602105A (en) * | 2019-09-17 | 2019-12-20 | 国家电网有限公司 | Large-scale parallelization network intrusion detection method based on k-means |
| CN111565171A (en) * | 2020-03-31 | 2020-08-21 | 北京三快在线科技有限公司 | Abnormal data detection method and device, electronic equipment and storage medium |
-
2020
- 2020-12-30 CN CN202011603971.7A patent/CN112671791B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102420723A (en) * | 2011-12-14 | 2012-04-18 | 南京邮电大学 | Anomaly detection method for various kinds of intrusion |
| US20170251012A1 (en) * | 2016-02-25 | 2017-08-31 | Darktrace Limited | Cyber security |
| CN110602105A (en) * | 2019-09-17 | 2019-12-20 | 国家电网有限公司 | Large-scale parallelization network intrusion detection method based on k-means |
| CN111565171A (en) * | 2020-03-31 | 2020-08-21 | 北京三快在线科技有限公司 | Abnormal data detection method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 杨永娇等: "基于Isolation Forest 和Random Forest 相结合的智能电网时间序列数据异常检测算法", 《计算机与现代化》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112671791B (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107154950B (en) | Method and system for detecting log stream abnormity | |
| US10785244B2 (en) | Anomaly detection method, learning method, anomaly detection device, and learning device | |
| CN111935170B (en) | Network abnormal flow detection method, device and equipment | |
| CN109450671B (en) | Log multi-combination alarm classification method and system | |
| CN111814664B (en) | Method, device, computer equipment and storage medium for identifying labels in drawing | |
| CN113254255B (en) | Cloud platform log analysis method, system, device and medium | |
| JP2019110513A (en) | Anomaly detection method, learning method, anomaly detection device, and learning device | |
| CN109995611B (en) | Traffic classification model establishing and traffic classification method, device, equipment and server | |
| CN109726737B (en) | Track-based abnormal behavior detection method and device | |
| CN110717551A (en) | Training method and device of flow identification model and electronic equipment | |
| CN106301979B (en) | Method and system for detecting abnormal channel | |
| CN112671791B (en) | Method, device, computer equipment and readable storage medium for detecting entity abnormity | |
| CN112003835B (en) | Security threat detection method and device, computer equipment and storage medium | |
| CN113282920A (en) | Log abnormity detection method and device, computer equipment and storage medium | |
| CN112463834A (en) | Method and device for automatically realizing root cause analysis in streaming processing and electronic equipment | |
| CN117294497A (en) | Network traffic abnormality detection method and device, electronic equipment and storage medium | |
| CN117221087A (en) | Alarm root cause positioning method, device and medium | |
| CN113535458B (en) | Abnormal false alarm processing method and device, storage medium and terminal | |
| CN113254672A (en) | Abnormal account identification method, system, equipment and readable storage medium | |
| CN115208938B (en) | User behavior control method and device and computer readable storage medium | |
| CN110138723A (en) | The determination method and system of malice community in a kind of mail network | |
| CN111368887B (en) | Training method of thunderstorm weather prediction model and thunderstorm weather prediction method | |
| CN115499187A (en) | API safety monitoring model training method, monitoring method, device and equipment | |
| CN113468011A (en) | Online data anomaly detection method and device, storage medium and equipment | |
| CN111368887A (en) | Training method of thunderstorm weather prediction model and thunderstorm weather prediction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 100097 2nd floor, building 1, yard 26, Xizhimenwai South Road, Xicheng District, Beijing Applicant after: Qianxin Wangshen information technology (Beijing) Co.,Ltd. Applicant after: Qianxin Technology Group Co.,Ltd. Address before: 100097 2nd floor, building 1, yard 26, Xizhimenwai South Road, Xicheng District, Beijing Applicant before: LEGENDSEC INFORMATION TECHNOLOGY (BEIJING) Inc. Applicant before: Qianxin Technology Group Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |