CN110830450A - Abnormal flow monitoring method, device and equipment based on statistics and storage medium - Google Patents
Abnormal flow monitoring method, device and equipment based on statistics and storage medium Download PDFInfo
- Publication number
- CN110830450A CN110830450A CN201910991150.6A CN201910991150A CN110830450A CN 110830450 A CN110830450 A CN 110830450A CN 201910991150 A CN201910991150 A CN 201910991150A CN 110830450 A CN110830450 A CN 110830450A
- Authority
- CN
- China
- Prior art keywords
- gaussian distribution
- user access
- data
- statistical
- abnormal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a statistical-based abnormal flow monitoring method, which comprises the following steps: collecting user access log records in a preset time period, cleaning and converting the user access log records to generate standard user access data; counting the distribution of statistical characteristics corresponding to standard user access data on different time dimensions respectively; mapping the distribution of the statistical characteristics on different time dimensions into corresponding multivariate Gaussian distribution and respectively carrying out parameter estimation; calculating Gaussian distribution probability values respectively corresponding to the statistical characteristics corresponding to the current network flow in each time dimension; judging whether the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow; and if so, judging that the current network flow is abnormal flow. The invention also discloses a device, equipment and a storage medium for monitoring the abnormal flow based on statistics. The invention is easy to deploy and low in implementation cost, and can flexibly deal with abnormal flow real-time alarm of different service scenes in different time periods.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a statistical-based abnormal flow monitoring method, device, equipment and storage medium.
Background
With the arrival of the information age, monitoring of abnormal network traffic is always an important ring in the field of information security. Network anomaly traffic refers to traffic in the network that varies significantly on an irregular basis. The method aims at sudden change abnormality which can occur in a short time of network traffic, and problems such as high-frequency operation, abnormal period access, file abnormality or access object abnormality and the like can exist behind the sudden change abnormality. Either type of problem may face quality of service degradation affecting normal user access and network security issues.
At present, abnormal flow monitoring is usually realized based on a machine learning mode, so that a corresponding technical system is required to be built and a monitoring model is required to be deployed, professional algorithm technicians are required to carry out operation and maintenance, and the abnormal flow monitoring is complex to realize and high in cost.
Disclosure of Invention
The invention mainly aims to provide a statistical-based abnormal traffic monitoring method, device, equipment and storage medium, and aims to solve the technical problems of complex deployment and high implementation cost of the conventional network abnormal traffic monitoring.
In order to achieve the above object, the present invention provides an abnormal traffic monitoring method based on statistics, wherein the abnormal traffic monitoring method comprises the following steps:
collecting user access log records in a preset time period based on a preset buried point;
cleaning and transforming the original data in the user access log record to generate standard user access data meeting the statistical requirement;
respectively sliding according to time windows corresponding to days, weeks and months, and respectively counting the distribution of statistical characteristics corresponding to standard user access data on different time dimensions;
mapping the distribution of the statistical characteristics on different time dimensions into corresponding multivariate Gaussian distributions, and respectively performing parameter estimation to obtain corresponding multivariate Gaussian distribution density functions;
calculating Gaussian distribution probability values respectively corresponding to statistical characteristics of user access log records corresponding to current network flow in each time dimension according to the multivariate Gaussian distribution density function;
judging whether the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow;
and if the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
Optionally, the mapping the distribution of the statistical features in different time dimensions to corresponding multivariate gaussian distributions and performing parameter estimation respectively to obtain corresponding multivariate gaussian distribution density functions includes:
respectively carrying out normalization processing or data transformation processing on the corresponding statistical characteristics in different time dimensions so as to map the distribution of the statistical characteristics in different time dimensions into corresponding multivariate Gaussian distribution;
taking the data corresponding to each multivariate Gaussian distribution as a sample, and solving a mean estimator and a covariance array estimator corresponding to each multivariate Gaussian distribution by adopting maximum likelihood estimation;
and generating a multi-element Gaussian distribution density function corresponding to each multi-element Gaussian distribution based on the mean value estimator and the covariance matrix estimator corresponding to each multi-element Gaussian distribution.
Optionally, the raw data set corresponding to the statistical features is normalized through the following formula:
wherein mu and sigma are respectively the mean value and variance of the original data set, and S is the normalized data;
and carrying out logarithmic transformation processing on the original data set corresponding to the statistical characteristics by the following formula:
y＝logc(1+λx)；
where x is the original data, y is the logarithmically transformed data, λ is set to 1, and c is set to the maximum value of the transformed data.
Optionally, the following function is adopted as the likelihood function corresponding to the P-ary gaussian distribution:
wherein, muAnd sigma are respectively a sample mean vector and a sample covariance matrix corresponding to P-element Gaussian distribution, XiThe ith statistical feature sample vector is represented, n represents n total statistical feature sample vectors, L represents a likelihood function, and f represents a probability density function.
Optionally, the cleaning and transforming the original data in the user access log record, and generating standard user access data meeting statistical requirements includes:
detecting whether the original data in the user access log record has a missing value or not;
if the missing value exists, calculating the missing value proportion corresponding to each field, and cleaning the missing value according to the missing value proportion and the field importance degree, wherein the missing value cleaning comprises the following steps: deleting the field of the missing value and completing the missing value by using an interpolation method;
sequencing original data in the user access log records, and calculating the similarity between each sequenced record and an adjacent record;
if the similarity between different records exceeds a preset threshold, judging to record repeatedly and deleting redundant data;
and transforming the cleaned data to generate user access standard data meeting the statistical requirement, wherein the transforming comprises the following steps: data type transformation, logarithmic transformation, data discretization.
Optionally, the user access log record comprises at least: user ID, user IP address, service party IP address, user access starting time, user access stopping time, user access ending time and access abnormal code; the statistical features at least include: user access amount, abnormal type, access time and whether the user is a new or reduced user;
the statistical characteristics are in the following format: the user IP address and the server IP address are used as user ID, and at least the user access amount, the abnormal type, the access time and whether the user is a new or reduced user are taken as specific characteristics.
Optionally, the determining whether the gaussian distribution probability value is smaller than a preset alarm threshold in a time dimension of the current network traffic includes:
drawing Gaussian distribution graphs of statistical characteristics of user access log records corresponding to network flow in each time dimension, and identifying contour lines corresponding to preset alarm thresholds on each Gaussian distribution graph;
judging whether the Gaussian distribution probability value corresponding to the current network flow is smaller than a preset alarm threshold value in the time dimension of the current network flow according to the corresponding Gaussian distribution curve graph in the time dimension of the current network flow;
and if the flow data is outside the contour line, judging that the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow.
Further, in order to achieve the above object, the present invention further provides an abnormal flow monitoring device based on statistics, wherein the abnormal flow monitoring device includes:
the collection module is used for collecting user access log records in a preset time period based on a preset buried point;
the standardized processing module is used for cleaning and converting the original data in the user access log record to generate standard user access data meeting the statistical requirement;
the statistical module is used for sliding according to time windows corresponding to days, weeks and months respectively, and counting the distribution of statistical characteristics corresponding to standard user access data on different time dimensions respectively;
the mapping module is used for mapping the distribution of the statistical characteristics on different time dimensions into corresponding multivariate Gaussian distribution and respectively carrying out parameter estimation to obtain corresponding multivariate Gaussian distribution density functions;
the calculation module is used for calculating Gaussian distribution probability values respectively corresponding to statistical characteristics of user access log records corresponding to the current network flow in each time dimension according to the multivariate Gaussian distribution density function;
the judging module is used for judging whether the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow; and if the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
Optionally, the mapping module includes:
the preprocessing unit is used for respectively carrying out normalization processing or data transformation processing on the corresponding statistical characteristics in different time dimensions so as to map the distribution of the statistical characteristics in different time dimensions into corresponding multivariate Gaussian distribution;
the estimation unit is used for solving mean value estimators and covariance matrix estimators corresponding to the multivariate Gaussian distributions by using the data corresponding to the multivariate Gaussian distributions as samples and adopting maximum likelihood estimation;
and the generating unit is used for generating a multi-element Gaussian distribution density function corresponding to each multi-element Gaussian distribution based on the mean value estimator and the covariance matrix estimator corresponding to each multi-element Gaussian distribution.
Optionally, the raw data set corresponding to the statistical features is normalized through the following formula:
wherein mu and sigma are respectively the mean value and variance of the original data set, and S is the normalized data;
and carrying out logarithmic transformation processing on the original data set corresponding to the statistical characteristics by the following formula:
y＝logc(1+λx)；
where x is the original data, y is the logarithmically transformed data, λ is set to 1, and c is set to the maximum value of the transformed data.
Optionally, the following function is adopted as the likelihood function corresponding to the P-ary gaussian distribution:
wherein, mu and sigma are divided intoSample mean vector and sample covariance matrix, X, corresponding to P-element Gaussian distributioniThe ith statistical feature sample vector is represented, n represents n total statistical feature sample vectors, L represents a likelihood function, and f represents a probability density function.
Optionally, the normalization processing module includes:
the cleaning unit is used for detecting whether the original data in the user access log record has a missing value or not; if the missing value exists, calculating the missing value proportion corresponding to each field, and cleaning the missing value according to the missing value proportion and the field importance degree, wherein the missing value cleaning comprises the following steps: deleting the field of the missing value and completing the missing value by using an interpolation method;
the sequencing unit is used for sequencing the original data in the user access log records and calculating the similarity between each sequenced record and the adjacent record; if the similarity between different records exceeds a preset threshold, judging to record repeatedly and deleting redundant data;
a transformation unit, configured to perform transformation processing on the cleaned data to generate user access standard data meeting the statistical requirement, where the transformation processing includes: data type transformation, logarithmic transformation, data discretization.
Optionally, the user access log record comprises at least: user ID, user IP address, service party IP address, user access starting time, user access stopping time, user access ending time and access abnormal code; the statistical features at least include: user access amount, abnormal type, access time and whether the user is a new or reduced user;
the statistical characteristics are in the following format: the user IP address and the server IP address are used as user ID, and at least the user access amount, the abnormal type, the access time and whether the user is a new or reduced user are taken as specific characteristics.
Optionally, the determining module is specifically configured to:
drawing Gaussian distribution graphs of statistical characteristics of user access log records corresponding to network flow in each time dimension, and identifying contour lines corresponding to preset alarm thresholds on each Gaussian distribution graph;
judging whether the Gaussian distribution probability value corresponding to the current network flow is smaller than a preset alarm threshold value in the time dimension of the current network flow according to the corresponding Gaussian distribution curve graph in the time dimension of the current network flow;
and if the flow data is outside the contour line, judging that the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow.
Further, to achieve the above object, the present invention further provides an abnormal traffic monitoring device based on statistics, where the abnormal traffic monitoring device includes a memory, a processor, and an abnormal traffic monitoring program stored in the memory and executable on the processor, and when the abnormal traffic monitoring program is executed by the processor, the abnormal traffic monitoring method implements the steps of any one of the above described abnormal traffic monitoring methods.
Further, in order to achieve the above object, the present invention also provides a computer readable storage medium, wherein the computer readable storage medium stores thereon an abnormal flow monitoring program, and when the abnormal flow monitoring program is executed by a processor, the abnormal flow monitoring program implements the steps of the abnormal flow monitoring method according to any one of the above.
The abnormal flow detection is carried out based on a statistical probability analysis method, statistical characteristic distribution corresponding to user access records is fitted into multivariate Gaussian distribution, and the abnormal flow detection is realized based on the characteristics of the multivariate Gaussian distribution. The method is short and bold, does not relate to complex algorithms, is easy to deploy and implement, can dynamically adjust the threshold value along with the real-time characteristics of the flow data, avoids the defect of inflexibility caused by regular alarm, and solves the problem of high implementation cost of complex algorithms based on machine learning and the like.
Drawings
Fig. 1 is a schematic structural diagram of an operating environment of an abnormal flow monitoring device according to an embodiment of the present invention;
FIG. 2 is a schematic flow chart illustrating an embodiment of a statistical-based abnormal traffic monitoring method according to the present invention;
FIG. 3 is a schematic diagram illustrating a detailed flow of step S40 in FIG. 2;
FIG. 4 is a schematic diagram illustrating a detailed flow of step S20 in FIG. 2;
fig. 5 is a schematic functional block diagram of an embodiment of the abnormal traffic monitoring apparatus according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides abnormal flow monitoring equipment based on statistics.
Referring to fig. 1, fig. 1 is a schematic structural diagram of an operating environment of an abnormal flow monitoring device according to an embodiment of the present invention.
As shown in fig. 1, the abnormal flow monitoring apparatus includes: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display (Display), an input unit such as a Keyboard (Keyboard), and the network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the hardware configuration of the abnormal flow monitoring device shown in fig. 1 does not constitute a limitation of the abnormal flow monitoring device, and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer-readable storage medium, may include therein an operating system, a network communication module, a user interface module, and a statistics-based abnormal traffic monitoring program. The operating system is a program for managing and controlling the abnormal flow monitoring equipment and software resources, and supports the operation of the abnormal flow monitoring program based on statistics and other software and/or programs.
In the hardware structure of the abnormal traffic monitoring apparatus shown in fig. 1, the network interface 1004 is mainly used for accessing a network; the user interface 1003 is mainly used for detecting a confirmation instruction, an editing instruction, and the like, and the processor 1001 may be configured to call a statistics-based abnormal traffic monitoring program stored in the memory 1005, and perform the following operations of the embodiments of the statistics-based abnormal traffic monitoring method.
Based on the above hardware structure of the abnormal traffic monitoring device, the present invention provides various embodiments of the abnormal traffic monitoring method based on statistics.
Referring to fig. 2, fig. 2 is a schematic flow chart of an embodiment of the abnormal traffic monitoring method based on statistics according to the present invention. In this embodiment, the abnormal traffic monitoring method includes the following steps:
step S10, collecting user access log records in a preset time period based on a preset buried point;
generally, network traffic has certain characteristics, and the characteristics conform to normal distribution, and the specific characteristics of the network traffic include user access time, user stay time, user access end time, access abnormal conditions, and the like. Therefore, in order to obtain the characteristics of the network traffic, in the embodiment, the log record data of the user access within the preset time period is collected through a preset buried point, such as a buried point in the log database. To more truly fit the characteristics of network traffic, it is therefore preferable to collect user access log records over a period of at least one month or more.
Optionally, in a specific embodiment, the user accessing the log record at least includes: user ID, user IP address, server IP address, user access start time, user access dwell time, user access end time, access exception code.
Step S20, cleaning and transforming the original data in the user access log record to generate standard user access data meeting the statistical requirement;
in this embodiment, for convenience of subsequent processing, the raw data in the collected user access log record is cleaned and transformed in advance, so as to generate standard user access data meeting the statistical requirements. The processing manner of cleaning and changing is not limited in this embodiment.
Data cleansing refers to filtering out unsatisfactory data, mainly including incomplete data, erroneous data and repeated data. In which, incomplete data, that is, some information that should be present, is missing, and such data needs to be removed or supplemented by interpolation processing. The wrong data refers to data with an incorrect format, such as an incorrect field format and an incorrect business meaning corresponding to the data. Duplicate data, such data needs to be culled.
The data conversion mainly converts inconsistent data, for example, uniform data of the same type of different service systems, for example, XX0001 for the code of the same supplier in the system a and YY0001 for the code in the system B, such data needs to be uniformly converted into the same code. In addition, the method also includes the calculation of the business rules, for example, different business systems have different business rules, and use different data indexes, and the indexes can be used after being calculated according to the corresponding business rules.
In this embodiment, the standard user access data obtained by cleaning and converting the original data in the user access log record is regarded as valid data, and can be used for subsequent statistical processing.
Step S30, respectively sliding according to time windows corresponding to the day, week and month, and respectively counting the distribution of statistical characteristics corresponding to standard user access data on different time dimensions;
in this embodiment, in order to monitor abnormal traffic more flexibly, a plurality of time dimensions are selected to count traffic characteristics, that is, the statistical characteristics corresponding to the statistical standard user access data are distributed in different time dimensions respectively.
Optionally, in a specific embodiment, the statistical characteristics at least include: user access amount, abnormal type, access time, whether a new user or a reduced user is provided, and the like; the statistics of the above features are calculated according to different time dimensions (day, week, and month), so as to obtain the distribution of the statistical features in different time dimensions. For example, during a day, access times are concentrated on: 9-12 am, 19 pm to 23 pm; during the week, monday through friday visits were small, while the saturday daily visits were large. Wherein, the new user is defined by comparing whether the user is existed before one day, one week or one month. If the user is the old user, otherwise, the user is the new user.
Optionally, in an embodiment, the statistical characteristics use the following format: and a combined key formed by the user IP address and the service party IP address is used as a user ID, and the contents including user access amount, abnormal type, access time, whether a new user or a reduced user and the like are used as specific characteristics.
Step S40, mapping the distribution of the statistical characteristics on different time dimensions into corresponding multivariate Gaussian distribution and respectively performing parameter estimation to obtain corresponding multivariate Gaussian distribution density functions;
in this embodiment, in order to fit the network traffic, it is necessary to fit the characteristics of the network traffic into corresponding multivariate gaussian distribution density functions, specifically, the distributions of the statistical characteristics corresponding to the same standard user access data in three time dimensions of day, week, and month are respectively mapped into corresponding multivariate gaussian distributions, and then parameter estimation is performed on each multivariate gaussian distribution to determine the characteristic parameter values in each multivariate gaussian distribution density function.
In this embodiment, it is preferable to adopt normalization processing or data transformation processing to induce the statistical distribution of the uniform samples, so that the data exhibits gaussian distribution characteristics. It should be further noted that the statistical characteristics in this embodiment need to be preset. When the statistical characteristics are set, if the selected statistical characteristics do not accord with Gaussian distribution, the data can be tried to be transformed by using various functions through the histogram distribution of the statistical data until the histogram distribution characteristics accord with the Gaussian distribution. In addition, if the currently selected statistical feature is not obvious enough, a statistical feature with more discrimination can be designed and added, so that the statistical feature distribution is more consistent with the gaussian distribution, for example, a plurality of associated features are combined into a new feature.
In this embodiment, after mapping the statistical feature distribution to the corresponding multivariate gaussian distribution, further parameter estimation is required to implement data fitting, so as to calculate feature parameter values of each multivariate gaussian distribution density function, such as a feature mean, a feature covariance, a quantile of probability distribution, and the like.
Step S50, calculating Gaussian distribution probability values respectively corresponding to statistical characteristics of user access log records corresponding to the current network flow in each time dimension according to the multivariate Gaussian distribution density function;
in this embodiment, the calculated multivariate gaussian distribution density function is as follows:
wherein the content of the first and second substances,
a vector of dimension D is represented as,
then it is the average of these vectors, and Σ represents all vectors
The covariance matrix of (2).
In this embodiment, the multiple gaussian distribution density function may be used to fit the gaussian distribution of the user access log in different time dimensions. Therefore, the Gaussian distribution probability values corresponding to the current user access log records in each time dimension can be calculated in real time through the multi-element Gaussian distribution density function, and due to the fact that the statistical feature distribution corresponding to the user access log records has the Gaussian distribution features, abnormal flow detection can be conducted through the Gaussian distribution probability values corresponding to the user access log records.
Step S60, judging whether the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow;
and step S70, if the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
In this embodiment, alarm thresholds corresponding to different time dimensions are set according to gaussian distribution characteristics and actual experience, for example, if the monitoring time dimension is one day, the alarm threshold is set to five thousandths, that is, data with a gaussian distribution probability value lower than five thousandths is used as alarm data; and if the monitoring time dimension is one week, setting the alarm threshold value to be three thousandths, namely, taking the data with the Gaussian distribution probability value lower than three thousandths as alarm data. And if the calculated Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
Optionally, in a specific embodiment, in order to facilitate visual monitoring of the network traffic, a gaussian distribution curve graph of statistical characteristics of user access log records corresponding to the network traffic in each time dimension may be drawn, and a contour line corresponding to an alarm threshold is identified on each gaussian distribution curve graph; judging whether the Gaussian distribution probability value corresponding to the current network flow is smaller than a preset alarm threshold value in the time dimension of the current network flow according to the corresponding Gaussian distribution curve graph in the time dimension of the current network flow; if the flow data appears within the contour line, the Gaussian distribution probability value is judged to be larger than a preset alarm threshold value in the time dimension of the current network flow, namely, the Gaussian distribution probability value is considered to be normal flow data, and if the flow data appears outside the contour line, the Gaussian distribution probability value is judged to be smaller than the preset alarm threshold value in the time dimension of the current network flow, namely, the Gaussian distribution probability value is considered to be abnormal flow data.
The embodiment performs abnormal flow detection based on a statistical probability analysis method, fits statistical characteristic distribution corresponding to user access records into multivariate Gaussian distribution, and realizes abnormal flow detection based on characteristics of the multivariate Gaussian distribution. The method is short and bold, does not relate to complex algorithms, is easy to deploy and implement, can dynamically adjust the threshold value along with the real-time characteristics of the flow data, avoids the defect of inflexibility caused by regular alarm, and solves the problem of high implementation cost of complex algorithms based on machine learning and the like.
In addition, in the embodiment, the user access log record data is respectively deployed according to different dimensionalities of day, week and month, so that the purpose of dynamically alarming according to different alarm thresholds in different time dimensions is achieved, the misinformation caused by different time dimensions is reduced, and meanwhile, due to the fact that a plurality of time dimensions are considered, the abnormal flow real-time alarm of different service scenes in different time periods can be flexibly responded.
Referring to fig. 3, fig. 3 is a schematic view of a detailed flow of the step S40 in fig. 2. Based on the foregoing embodiment, in this embodiment, the foregoing step S40 further includes:
step S401, respectively carrying out normalization processing or data transformation processing on the corresponding statistical characteristics in different time dimensions so as to map the distribution of the statistical characteristics in different time dimensions into corresponding multivariate Gaussian distribution;
in this embodiment, it is considered that the data may not directly satisfy the gaussian distribution, and therefore some processing needs to be performed on the data so that the data satisfies the gaussian distribution. In this embodiment, preferably, the statistical characteristic distribution obtained by statistics is mapped to a corresponding multivariate gaussian distribution by normalization processing or data transformation processing.
(1) And normalizing the original data set corresponding to the statistical characteristics by the following formula:
wherein mu and sigma are respectively the mean value and variance of the original data set, and S is the normalized data;
(2) and carrying out logarithmic transformation processing on the original data set corresponding to the statistical characteristics by the following formula:
y＝logc(1+λx)；
wherein mu and sigma are respectively the mean value and variance of the original data set, and S is the normalized data; by carrying out logarithmic transformation on the original data, the values of the original dense intervals are dispersed as much as possible, and the values of the original dispersed intervals are aggregated as much as possible, so that the data distribution is close to normal distribution, and the data is unrelated to the average value of the distribution.
In this embodiment, the data exhibits a gaussian distribution characteristic through normalization processing or data transformation processing, and then the data can be fitted to the gaussian distribution to facilitate abnormal flow monitoring.
Step S402, taking the data corresponding to each multivariate Gaussian distribution as a sample, and solving the mean value estimator and the covariance matrix estimator corresponding to each multivariate Gaussian distribution by adopting maximum likelihood estimation;
suppose X(1)，X(2)，...，X(n)Respectively mapped to P-element Gaussian distribution N for statistical characteristicspThe samples corresponding to (mu, sigma) construct likelihood functions corresponding to multiple Gaussian distributions, namely:
mu and sigma are sample mean vectors and sample covariance arrays corresponding to P-element Gaussian distribution respectively, Xi represents the ith statistical characteristic sample vector, n represents n statistical characteristic sample vectors, L represents a likelihood function, and f represents a probability density function.
To find the estimates of μ and Σ that make the above equation extreme, the logarithm is taken on both sides of the above equation, resulting in the following equation:
since the logarithmic function is a strictly monotonic increasing function, estimates of μ and Σ can be obtained by maximizing lnL (μ, Σ). Therefore, partial derivatives are obtained for μ and Σ for the above log-likelihood functions, respectively, to obtain the following equations:
the above equation is calculated to obtain maximum likelihood estimators for μ and Σ:
from the above, the mean estimator of multivariate gaussian distribution is a sample mean vector, and the covariance matrix estimator is a sample covariance matrix.
Step S403, generating a multivariate gaussian distribution density function corresponding to each multivariate gaussian distribution based on the mean estimator and the covariance matrix estimator corresponding to each multivariate gaussian distribution.
In this embodiment, after obtaining the parameter estimator of each multivariate gaussian distribution, a multivariate gaussian distribution density function corresponding to each multivariate gaussian distribution can be generated based on the respective sample data of each multivariate gaussian distribution. For example, based on sample data in one day, a sample mean vector and a sample covariance matrix corresponding to the multivariate gaussian distribution are calculated, so as to generate a multivariate gaussian distribution density function corresponding to the time dimension of the day.
According to the method and the device, the user access log record data are respectively deployed according to different dimensionalities of day, week and month, so that the purpose of dynamically alarming according to different alarming thresholds in different time dimensionalities is achieved, the misinformation caused by different time dimensionalities is reduced, and meanwhile due to the fact that a plurality of time dimensionalities are considered, the abnormal flow real-time alarming of different service scenes in different time periods can be flexibly handled.
Referring to fig. 4, fig. 4 is a schematic view of a detailed flow of the step S20 in fig. 2. Based on the foregoing embodiment, in this embodiment, the foregoing step S20 further includes:
step S201, detecting whether the original data in the user access log record has a missing value;
in this embodiment, the user access log records a plurality of information, such as a user ID, user and server IP addresses, user access time, user dwell time, user access end time, access exception condition, access state, exception type code, and exception type description, using a plurality of fields, and if a field corresponding to a record has a missing value, it is determined that the record has a missing value.
Step S202, if a missing value exists, calculating the missing value proportion corresponding to each field, and cleaning the missing value according to the missing value proportion and the field importance degree, wherein the cleaning of the missing value comprises the following steps: deleting the field of the missing value and completing the missing value by using an interpolation method;
in this embodiment, if there is a missing value in one or some fields in the user access log record, the proportion of the missing value corresponding to each field, for example, there are 100 user access log records, and if there is a missing value in 10 records corresponding to a field, the proportion of the missing value corresponding to the field is 10%.
In this embodiment, the importance levels of different fields in the actual application scenario are different. For example, the user IP address is more important than the server IP address, and the user access time is more important than the user dwell time. The different degrees of importance of the fields differ in the cleaning strategy used. For example, if the missing value ratio is high and the field importance level is low, the missing value field is deleted as it is, and if the missing value ratio is low and the field importance level is high, the missing value is completed by interpolation.
Step S203, sequencing the original data in the user access log records, and calculating the similarity between each sequenced record and the adjacent record;
step S204, if the similarity between different records exceeds a preset threshold, determining to repeatedly record and delete redundant data;
in this embodiment, the duplicate records are further subjected to deduplication processing, specifically, all original data in the log records accessed by the user are sorted first, for example, sorted based on the numerical value of a certain field, for example, sorted based on access time, and then the similarity between each sorted record and an adjacent record is calculated, for example, the similarity between different records is calculated by adopting a field matching algorithm, a standardized euclidean distance, and the like. If the similarity between different records exceeds a preset threshold (such as 90%), the recording is determined to be repeated and redundant data is deleted.
Step S205, performing transformation processing on the cleaned data to generate user access standard data meeting the statistical requirement, where the transformation processing includes: data type transformation, logarithmic transformation, data discretization.
In this embodiment, in order to make the data more convenient for statistics, the cleaned data is further transformed, and then the user access standard data meeting the statistical requirements is generated.
A. Converting data types, such as floating point type data to integer type data, so as to facilitate calculation;
B. the original data is subjected to logarithmic transformation, so that the values of the original dense intervals are dispersed as much as possible, and the values of the original dispersed intervals are aggregated as much as possible, so that the data distribution is close to normal distribution, and the data is unrelated to the average value of the distribution.
C. Discretizing the continuous data, such as time, so that the interval can be used to analyze the characteristics of the data, for example, performing equal-width discretization on the continuous data, such as dividing the time into morning, noon, afternoon, evening, and midnight.
In this embodiment, through rinsing and transform processing to data to obtain the standard data that accords with the statistical requirement, not only be convenient for statistical analysis, can also further promote flow monitoring's accuracy simultaneously.
The invention also provides an abnormal flow monitoring device based on statistics.
Referring to fig. 5, fig. 5 is a functional module diagram of an embodiment of the abnormal flow monitoring device based on statistics according to the present invention. In this embodiment, the abnormal flow monitoring apparatus includes:
the collection module 10 is configured to collect user access log records within a preset time period based on a preset buried point;
the standardized processing module 20 is used for cleaning and transforming the original data in the user access log record to generate standard user access data meeting the statistical requirements;
the statistical module 30 is configured to slide according to time windows corresponding to days, weeks, and months, respectively, and perform statistics on distribution of statistical characteristics corresponding to standard user access data in different time dimensions, respectively;
the mapping module 40 is configured to map the distribution of the statistical characteristics in different time dimensions to corresponding multivariate gaussian distributions and perform parameter estimation respectively to obtain corresponding multivariate gaussian distribution density functions;
a calculating module 50, configured to calculate, according to the multivariate gaussian distribution density function, gaussian distribution probability values respectively corresponding to statistical characteristics of user access log records corresponding to current network traffic in each time dimension;
a judging module 60, configured to judge whether the gaussian distribution probability value is smaller than a preset alarm threshold in a time dimension of the current network traffic; and if the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
Optionally, the mapping module 40 includes:
the preprocessing unit is used for respectively carrying out normalization processing or data transformation processing on the corresponding statistical characteristics in different time dimensions so as to map the distribution of the statistical characteristics in different time dimensions into corresponding multivariate Gaussian distribution;
the estimation unit is used for solving mean value estimators and covariance matrix estimators corresponding to the multivariate Gaussian distributions by using the data corresponding to the multivariate Gaussian distributions as samples and adopting maximum likelihood estimation;
and the generating unit is used for generating a multi-element Gaussian distribution density function corresponding to each multi-element Gaussian distribution based on the mean value estimator and the covariance matrix estimator corresponding to each multi-element Gaussian distribution.
Optionally, the raw data set corresponding to the statistical features is normalized through the following formula:
wherein mu and sigma are respectively the mean value and variance of the original data set, and S is the normalized data;
and carrying out logarithmic transformation processing on the original data set corresponding to the statistical characteristics by the following formula:
y＝logc(1+λx)；
where x is the original data, y is the logarithmically transformed data, λ is set to 1, and c is set to the maximum value of the transformed data.
Optionally, the following function is adopted as the likelihood function corresponding to the P-ary gaussian distribution:
wherein mu and sigma are respectively a sample mean vector and a sample covariance matrix corresponding to P-element Gaussian distribution, XiThe ith statistical feature sample vector is represented, n represents n total statistical feature sample vectors, L represents a likelihood function, and f represents a probability density function.
Optionally, the normalization processing module 20 includes:
the cleaning unit is used for detecting whether the original data in the user access log record has a missing value or not; if the missing value exists, calculating the missing value proportion corresponding to each field, and cleaning the missing value according to the missing value proportion and the field importance degree, wherein the missing value cleaning comprises the following steps: deleting the field of the missing value and completing the missing value by using an interpolation method;
the sequencing unit is used for sequencing the original data in the user access log records and calculating the similarity between each sequenced record and the adjacent record; if the similarity between different records exceeds a preset threshold, judging to record repeatedly and deleting redundant data;
a transformation unit, configured to perform transformation processing on the cleaned data to generate user access standard data meeting the statistical requirement, where the transformation processing includes: data type transformation, logarithmic transformation, data discretization.
Optionally, the user access log record comprises at least: user ID, user IP address, service party IP address, user access starting time, user access stopping time, user access ending time and access abnormal code; the statistical features at least include: user access amount, abnormal type, access time and whether the user is a new or reduced user;
the statistical characteristics are in the following format: the user IP address and the server IP address are used as user ID, and at least the user access amount, the abnormal type, the access time and whether the user is a new or reduced user are taken as specific characteristics.
Optionally, the determining module 60 is specifically configured to:
drawing Gaussian distribution graphs of statistical characteristics of user access log records corresponding to network flow in each time dimension, and identifying contour lines corresponding to preset alarm thresholds on each Gaussian distribution graph;
judging whether the Gaussian distribution probability value corresponding to the current network flow is smaller than a preset alarm threshold value in the time dimension of the current network flow according to the corresponding Gaussian distribution curve graph in the time dimension of the current network flow;
and if the flow data is outside the contour line, judging that the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow.
Based on the same description of the embodiment as the abnormal traffic monitoring method of the present invention, the contents of the embodiment of the abnormal traffic monitoring apparatus are not described in detail in this embodiment.
The embodiment performs abnormal flow detection based on a statistical probability analysis method, fits statistical characteristic distribution corresponding to user access records into multivariate Gaussian distribution, and realizes abnormal flow detection based on characteristics of the multivariate Gaussian distribution. The method is short and bold, does not relate to complex algorithms, is easy to deploy and implement, can dynamically adjust the threshold value along with the real-time characteristics of the flow data, avoids the defect of inflexibility caused by regular alarm, and solves the problem of high implementation cost of complex algorithms based on machine learning and the like.
In addition, in the embodiment, the user access log record data is respectively deployed according to different dimensionalities of day, week and month, so that the purpose of dynamically alarming according to different alarm thresholds in different time dimensions is achieved, the misinformation caused by different time dimensions is reduced, and meanwhile, due to the fact that a plurality of time dimensions are considered, the abnormal flow real-time alarm of different service scenes in different time periods can be flexibly responded.
The invention also provides a computer readable storage medium.
In this embodiment, the computer readable storage medium stores an abnormal traffic monitoring program, and the abnormal traffic monitoring program, when executed by the processor, implements the steps of the statistics-based abnormal traffic monitoring method as described in any one of the above embodiments. The method implemented when the abnormal traffic monitoring program is executed by the processor may refer to each embodiment of the statistical-based abnormal traffic monitoring method of the present invention, and therefore, the description thereof is not repeated.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM), and includes instructions for causing a terminal (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The present invention is described in connection with the accompanying drawings, but the present invention is not limited to the above embodiments, which are only illustrative and not restrictive, and those skilled in the art can make various changes without departing from the spirit and scope of the invention as defined by the appended claims, and all changes that come within the meaning and range of equivalency of the specification and drawings that are obvious from the description and the attached claims are intended to be embraced therein.
Claims (10)
1. An abnormal flow monitoring method based on statistics is characterized by comprising the following steps:
collecting user access log records in a preset time period based on a preset buried point;
cleaning and transforming the original data in the user access log record to generate standard user access data meeting the statistical requirement;
respectively sliding according to time windows corresponding to days, weeks and months, and respectively counting the distribution of statistical characteristics corresponding to standard user access data on different time dimensions;
mapping the distribution of the statistical characteristics on different time dimensions into corresponding multivariate Gaussian distributions, and respectively performing parameter estimation to obtain corresponding multivariate Gaussian distribution density functions;
calculating Gaussian distribution probability values respectively corresponding to statistical characteristics of user access log records corresponding to current network flow in each time dimension according to the multivariate Gaussian distribution density function;
judging whether the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow;
and if the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
2. The abnormal flow monitoring method based on statistics as claimed in claim 1, wherein the mapping the distribution of the statistical characteristics in different time dimensions to corresponding multivariate gaussian distributions and performing parameter estimation respectively to obtain corresponding multivariate gaussian distribution density functions comprises:
respectively carrying out normalization processing or data transformation processing on the corresponding statistical characteristics in different time dimensions so as to map the distribution of the statistical characteristics in different time dimensions into corresponding multivariate Gaussian distribution;
taking the data corresponding to each multivariate Gaussian distribution as a sample, and solving a mean estimator and a covariance array estimator corresponding to each multivariate Gaussian distribution by adopting maximum likelihood estimation;
and generating a multi-element Gaussian distribution density function corresponding to each multi-element Gaussian distribution based on the mean value estimator and the covariance matrix estimator corresponding to each multi-element Gaussian distribution.
3. The method of claim 2, wherein the raw data set corresponding to the statistical features is normalized by the following formula:
wherein mu and sigma are respectively the mean value and variance of the original data set, and S is the normalized data;
and carrying out logarithmic transformation processing on the original data set corresponding to the statistical characteristics by the following formula:
y＝logc(1+λx)；
where x is the original data, y is the logarithmically transformed data, λ is set to 1, and c is set to the maximum value of the transformed data.
4. A method as claimed in claim 2 or 3, wherein the following function is used as the likelihood function for the P-ary gaussian distribution:
wherein mu and sigma are respectively a sample mean vector and a sample covariance matrix corresponding to P-element Gaussian distribution, XiThe ith statistical feature sample vector is represented, n represents n total statistical feature sample vectors, L represents a likelihood function, and f represents a probability density function.
5. The method of claim 1, wherein the cleaning and transforming the raw data in the user access log records to generate standard user access data meeting statistical requirements comprises:
detecting whether the original data in the user access log record has a missing value or not;
if the missing value exists, calculating the missing value proportion corresponding to each field, and cleaning the missing value according to the missing value proportion and the field importance degree, wherein the missing value cleaning comprises the following steps: deleting the field of the missing value and completing the missing value by using an interpolation method;
sequencing original data in the user access log records, and calculating the similarity between each sequenced record and an adjacent record;
if the similarity between different records exceeds a preset threshold, judging to record repeatedly and deleting redundant data;
and transforming the cleaned data to generate user access standard data meeting the statistical requirement, wherein the transforming comprises the following steps: data type transformation, logarithmic transformation, data discretization.
6. The statistics-based abnormal traffic monitoring method of claim 1, wherein the user accessing the log record comprises at least: user ID, user IP address, service party IP address, user access starting time, user access stopping time, user access ending time and access abnormal code; the statistical features at least include: user access amount, abnormal type, access time and whether the user is a new or reduced user;
the statistical characteristics are in the following format: the user IP address and the server IP address are used as user ID, and at least the user access amount, the abnormal type, the access time and whether the user is a new or reduced user are taken as specific characteristics.
7. The abnormal traffic monitoring method based on statistics as claimed in claim 1, 5 or 6, wherein the determining whether the gaussian distribution probability value is smaller than the preset alarm threshold in the time dimension of the current network traffic comprises:
drawing Gaussian distribution graphs of statistical characteristics of user access log records corresponding to network flow in each time dimension, and identifying contour lines corresponding to preset alarm thresholds on each Gaussian distribution graph;
judging whether the Gaussian distribution probability value corresponding to the current network flow is smaller than a preset alarm threshold value in the time dimension of the current network flow according to the corresponding Gaussian distribution curve graph in the time dimension of the current network flow;
and if the flow data is outside the contour line, judging that the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow.
8. An abnormal flow monitoring device based on statistics, the abnormal flow monitoring device comprising:
the collection module is used for collecting user access log records in a preset time period based on a preset buried point;
the standardized processing module is used for cleaning and converting the original data in the user access log record to generate standard user access data meeting the statistical requirement;
the statistical module is used for sliding according to time windows corresponding to days, weeks and months respectively, and counting the distribution of statistical characteristics corresponding to standard user access data on different time dimensions respectively;
the mapping module is used for mapping the distribution of the statistical characteristics on different time dimensions into corresponding multivariate Gaussian distribution and respectively carrying out parameter estimation to obtain corresponding multivariate Gaussian distribution density functions;
the calculation module is used for calculating Gaussian distribution probability values respectively corresponding to statistical characteristics of user access log records corresponding to the current network flow in each time dimension according to the multivariate Gaussian distribution density function;
the judging module is used for judging whether the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow; and if the Gaussian distribution probability value is smaller than a preset alarm threshold value in the time dimension of the current network flow, judging the current network flow to be abnormal flow.
9. A statistics-based abnormal traffic monitoring device, characterized in that the abnormal traffic monitoring device comprises a memory, a processor and an abnormal traffic monitoring program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the statistics-based abnormal traffic monitoring method according to any of claims 1-7.
10. A computer readable storage medium, having stored thereon an abnormal traffic monitoring program, which when executed by a processor, performs the steps of the statistics-based abnormal traffic monitoring method of any of claims 1-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910991150.6A CN110830450A (en) | 2019-10-18 | 2019-10-18 | Abnormal flow monitoring method, device and equipment based on statistics and storage medium |
| PCT/CN2020/093392 WO2021073114A1 (en) | 2019-10-18 | 2020-05-29 | Abnormal traffic monitoring method, apparatus and device based on statistics, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910991150.6A CN110830450A (en) | 2019-10-18 | 2019-10-18 | Abnormal flow monitoring method, device and equipment based on statistics and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110830450A true CN110830450A (en) | 2020-02-21 |
Family
ID=69549556
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910991150.6A Pending CN110830450A (en) | 2019-10-18 | 2019-10-18 | Abnormal flow monitoring method, device and equipment based on statistics and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110830450A (en) |
| WO (1) | WO2021073114A1 (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111447228A (en) * | 2020-03-27 | 2020-07-24 | 四川虹美智能科技有限公司 | Intelligent household appliance access request processing method and system, cloud server and intelligent air conditioner |
| CN112148747A (en) * | 2020-09-08 | 2020-12-29 | 银清科技有限公司 | Transaction system log analysis method and device based on R language |
| WO2021073114A1 (en) * | 2019-10-18 | 2021-04-22 | 平安科技（深圳）有限公司 | Abnormal traffic monitoring method, apparatus and device based on statistics, and storage medium |
| CN112818244A (en) * | 2021-02-24 | 2021-05-18 | 深圳市网联安瑞网络科技有限公司 | Method, system and terminal for judging activity of channel, group and group user |
| CN113271322A (en) * | 2021-07-20 | 2021-08-17 | 北京明略软件系统有限公司 | Abnormal flow detection method and device, electronic equipment and storage medium |
| CN113542236A (en) * | 2021-06-28 | 2021-10-22 | 中孚安全技术有限公司 | Abnormal user detection method based on kernel density estimation and exponential smoothing algorithm |
| CN113852591A (en) * | 2021-06-08 | 2021-12-28 | 天翼智慧家庭科技有限公司 | Camera abnormal access identification and alarm method based on improved quartile bit difference method |
| CN114079624A (en) * | 2022-01-18 | 2022-02-22 | 广东道一信息技术股份有限公司 | Architecture data flow monitoring method and system based on multi-user access |
| CN114692058A (en) * | 2022-06-01 | 2022-07-01 | 天津市普迅电力信息技术有限公司 | Automatic point burying method and system based on VUE (virtual environment) architecture and electronic equipment |
| CN114745328A (en) * | 2022-02-16 | 2022-07-12 | 多点生活(成都)科技有限公司 | Dynamic gateway current limiting method and real-time current limiting method formed by same |
| CN115174254A (en) * | 2022-07-22 | 2022-10-11 | 科来网络技术股份有限公司 | Flow abnormity warning method and device, electronic equipment and storage medium |
| CN117290380A (en) * | 2023-11-14 | 2023-12-26 | 华青融天(北京)软件股份有限公司 | Abnormal dimension data generation method, device, equipment and computer readable medium |
| CN117527611A (en) * | 2023-11-07 | 2024-02-06 | 北京太极信息系统技术有限公司 | Gaussian distribution-based fault dynamic prediction method and related device |
| JP7441291B1 (en) | 2022-10-18 | 2024-02-29 | 財団法人 資訊工業策進会 | Information security early warning device and method |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113225339B (en) * | 2021-05-07 | 2023-04-07 | 恒安嘉新（北京）科技股份公司 | Network security monitoring method and device, computer equipment and storage medium |
| CN115174358B (en) * | 2022-09-08 | 2023-01-17 | 浪潮电子信息产业股份有限公司 | Monitoring processing method, system, equipment and storage medium for storage cluster interface |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105243301A (en) * | 2014-07-09 | 2016-01-13 | 阿里巴巴集团控股有限公司 | Keyboard input abnormality detection method and apparatus as well as security prompt method and apparatus |
| CN105516127A (en) * | 2015-12-07 | 2016-04-20 | 中国科学院信息工程研究所 | Internal threat detection-oriented user cross-domain behavior pattern mining method |
| CN107086944A (en) * | 2017-06-22 | 2017-08-22 | 北京奇艺世纪科技有限公司 | A kind of method for detecting abnormality and device |
| CN107168854A (en) * | 2017-06-01 | 2017-09-15 | 北京京东尚科信息技术有限公司 | Detection method, device, equipment and readable storage medium storing program for executing are clicked in Internet advertising extremely |
| CN107222472A (en) * | 2017-05-26 | 2017-09-29 | 电子科技大学 | A kind of user behavior method for detecting abnormality under Hadoop clusters |
| CN107302547A (en) * | 2017-08-21 | 2017-10-27 | 深信服科技股份有限公司 | A kind of web service exceptions detection method and device |
| CN107341095A (en) * | 2017-06-27 | 2017-11-10 | 北京优特捷信息技术有限公司 | A kind of method and device of intellectual analysis daily record data |
| CN107483455A (en) * | 2017-08-25 | 2017-12-15 | 国家计算机网络与信息安全管理中心 | A kind of network node abnormality detection method and system based on stream |
| CN108234463A (en) * | 2017-12-22 | 2018-06-29 | 杭州安恒信息技术有限公司 | A kind of consumer's risk assessment and analysis method based on multidimensional behavior model |
| CN109614576A (en) * | 2018-12-11 | 2019-04-12 | 福建工程学院 | Transformer exception detection method based on Multi-dimensional Gaussian distribution and trend segmentation |
| US20190158522A1 (en) * | 2018-01-02 | 2019-05-23 | Maryam AMIRMAZLAGHANI | Generalized likelihood ratio test (glrt) based network intrusion detection system in wavelet domain |
| CN109960631A (en) * | 2019-03-19 | 2019-07-02 | 山东九州信泰信息科技股份有限公司 | A kind of detecting real-time method of security incident exception |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102014031A (en) * | 2010-12-31 | 2011-04-13 | 湖南神州祥网科技有限公司 | Method and system for network flow anomaly detection |
| CN107154950B (en) * | 2017-07-24 | 2021-05-04 | 深信服科技股份有限公司 | Method and system for detecting log stream abnormity |
| CN107370766B (en) * | 2017-09-07 | 2020-09-11 | 杭州安恒信息技术股份有限公司 | Network flow abnormity detection method and system |
| CN110830450A (en) * | 2019-10-18 | 2020-02-21 | 平安科技（深圳）有限公司 | Abnormal flow monitoring method, device and equipment based on statistics and storage medium |
-
2019
- 2019-10-18 CN CN201910991150.6A patent/CN110830450A/en active Pending
-
2020
- 2020-05-29 WO PCT/CN2020/093392 patent/WO2021073114A1/en active Application Filing
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105243301A (en) * | 2014-07-09 | 2016-01-13 | 阿里巴巴集团控股有限公司 | Keyboard input abnormality detection method and apparatus as well as security prompt method and apparatus |
| CN105516127A (en) * | 2015-12-07 | 2016-04-20 | 中国科学院信息工程研究所 | Internal threat detection-oriented user cross-domain behavior pattern mining method |
| CN107222472A (en) * | 2017-05-26 | 2017-09-29 | 电子科技大学 | A kind of user behavior method for detecting abnormality under Hadoop clusters |
| CN107168854A (en) * | 2017-06-01 | 2017-09-15 | 北京京东尚科信息技术有限公司 | Detection method, device, equipment and readable storage medium storing program for executing are clicked in Internet advertising extremely |
| CN107086944A (en) * | 2017-06-22 | 2017-08-22 | 北京奇艺世纪科技有限公司 | A kind of method for detecting abnormality and device |
| CN107341095A (en) * | 2017-06-27 | 2017-11-10 | 北京优特捷信息技术有限公司 | A kind of method and device of intellectual analysis daily record data |
| CN107302547A (en) * | 2017-08-21 | 2017-10-27 | 深信服科技股份有限公司 | A kind of web service exceptions detection method and device |
| CN107483455A (en) * | 2017-08-25 | 2017-12-15 | 国家计算机网络与信息安全管理中心 | A kind of network node abnormality detection method and system based on stream |
| CN108234463A (en) * | 2017-12-22 | 2018-06-29 | 杭州安恒信息技术有限公司 | A kind of consumer's risk assessment and analysis method based on multidimensional behavior model |
| US20190158522A1 (en) * | 2018-01-02 | 2019-05-23 | Maryam AMIRMAZLAGHANI | Generalized likelihood ratio test (glrt) based network intrusion detection system in wavelet domain |
| CN109614576A (en) * | 2018-12-11 | 2019-04-12 | 福建工程学院 | Transformer exception detection method based on Multi-dimensional Gaussian distribution and trend segmentation |
| CN109960631A (en) * | 2019-03-19 | 2019-07-02 | 山东九州信泰信息科技股份有限公司 | A kind of detecting real-time method of security incident exception |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021073114A1 (en) * | 2019-10-18 | 2021-04-22 | 平安科技（深圳）有限公司 | Abnormal traffic monitoring method, apparatus and device based on statistics, and storage medium |
| CN111447228A (en) * | 2020-03-27 | 2020-07-24 | 四川虹美智能科技有限公司 | Intelligent household appliance access request processing method and system, cloud server and intelligent air conditioner |
| CN112148747A (en) * | 2020-09-08 | 2020-12-29 | 银清科技有限公司 | Transaction system log analysis method and device based on R language |
| CN112818244A (en) * | 2021-02-24 | 2021-05-18 | 深圳市网联安瑞网络科技有限公司 | Method, system and terminal for judging activity of channel, group and group user |
| CN113852591A (en) * | 2021-06-08 | 2021-12-28 | 天翼智慧家庭科技有限公司 | Camera abnormal access identification and alarm method based on improved quartile bit difference method |
| CN113852591B (en) * | 2021-06-08 | 2023-09-22 | 天翼数字生活科技有限公司 | Camera abnormal access identification and alarm method based on improved four-level difference method |
| CN113542236A (en) * | 2021-06-28 | 2021-10-22 | 中孚安全技术有限公司 | Abnormal user detection method based on kernel density estimation and exponential smoothing algorithm |
| CN113271322A (en) * | 2021-07-20 | 2021-08-17 | 北京明略软件系统有限公司 | Abnormal flow detection method and device, electronic equipment and storage medium |
| CN113271322B (en) * | 2021-07-20 | 2021-11-23 | 北京明略软件系统有限公司 | Abnormal flow detection method and device, electronic equipment and storage medium |
| CN114079624A (en) * | 2022-01-18 | 2022-02-22 | 广东道一信息技术股份有限公司 | Architecture data flow monitoring method and system based on multi-user access |
| CN114745328A (en) * | 2022-02-16 | 2022-07-12 | 多点生活(成都)科技有限公司 | Dynamic gateway current limiting method and real-time current limiting method formed by same |
| CN114745328B (en) * | 2022-02-16 | 2023-12-26 | 多点生活(成都)科技有限公司 | Gateway dynamic current limiting method and real-time current limiting method formed by same |
| CN114692058B (en) * | 2022-06-01 | 2022-08-02 | 天津市普迅电力信息技术有限公司 | Automatic point burying method and system based on VUE (virtual environment) architecture and electronic equipment |
| CN114692058A (en) * | 2022-06-01 | 2022-07-01 | 天津市普迅电力信息技术有限公司 | Automatic point burying method and system based on VUE (virtual environment) architecture and electronic equipment |
| CN115174254A (en) * | 2022-07-22 | 2022-10-11 | 科来网络技术股份有限公司 | Flow abnormity warning method and device, electronic equipment and storage medium |
| CN115174254B (en) * | 2022-07-22 | 2023-10-31 | 科来网络技术股份有限公司 | Flow abnormality warning method and device, electronic equipment and storage medium |
| JP7441291B1 (en) | 2022-10-18 | 2024-02-29 | 財団法人 資訊工業策進会 | Information security early warning device and method |
| CN117527611A (en) * | 2023-11-07 | 2024-02-06 | 北京太极信息系统技术有限公司 | Gaussian distribution-based fault dynamic prediction method and related device |
| CN117527611B (en) * | 2023-11-07 | 2024-06-07 | 北京太极信息系统技术有限公司 | Gaussian distribution-based fault dynamic prediction method, system, electronic equipment and storage medium |
| CN117290380A (en) * | 2023-11-14 | 2023-12-26 | 华青融天(北京)软件股份有限公司 | Abnormal dimension data generation method, device, equipment and computer readable medium |
| CN117290380B (en) * | 2023-11-14 | 2024-02-06 | 华青融天(北京)软件股份有限公司 | Abnormal dimension data generation method, device, equipment and computer readable medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021073114A1 (en) | 2021-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110830450A (en) | Abnormal flow monitoring method, device and equipment based on statistics and storage medium | |
| CN110839016B (en) | Abnormal flow monitoring method, device, equipment and storage medium | |
| US9471544B1 (en) | Anomaly detection in a signal | |
| WO2017076154A1 (en) | Method and apparatus for predicting network event and establishing network event prediction model | |
| CN111796957B (en) | Transaction abnormal root cause analysis method and system based on application log | |
| CN111176953B (en) | Abnormality detection and model training method, computer equipment and storage medium | |
| CN110647447B (en) | Abnormal instance detection method, device, equipment and medium for distributed system | |
| CN111800389A (en) | Port network intrusion detection method based on Bayesian network | |
| CN113723861A (en) | Abnormal electricity consumption behavior detection method and device, computer equipment and storage medium | |
| CN112416590A (en) | Server system resource adjusting method and device, computer equipment and storage medium | |
| CN112446601A (en) | Data diagnosis method and system for non-computable region | |
| CN117130851A (en) | High-performance computing cluster operation efficiency evaluation method and system | |
| CN113535458B (en) | Abnormal false alarm processing method and device, storage medium and terminal | |
| CN114785616A (en) | Data risk detection method and device, computer equipment and storage medium | |
| CN112422333B (en) | Distribution network condition determining method, system and related device | |
| CN116016115A (en) | Method, device, equipment, medium and program product for monitoring flow of network line | |
| CN112988536B (en) | Data anomaly detection method, device, equipment and storage medium | |
| CN113569879B (en) | Training method of abnormal recognition model, abnormal account recognition method and related device | |
| CN112100037B (en) | Alarm level identification method, device, electronic equipment and storage medium | |
| CN115169089A (en) | Wind power probability prediction method and device based on kernel density estimation and copula | |
| CN112651019A (en) | Abnormal behavior detection method and device based on unsupervised learning and computing terminal | |
| CN112417007A (en) | Data analysis method and device, electronic equipment and storage medium | |
| CN111798237A (en) | Abnormal transaction diagnosis method and system based on application log | |
| CN113537363B (en) | Abnormal object detection method and device, electronic equipment and storage medium | |
| CN115277185B (en) | Operation and maintenance system anomaly detection method based on graph neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200221 |