CN112653589A - Network data flow abnormity detection method based on host data flow characteristic extraction - Google Patents
Network data flow abnormity detection method based on host data flow characteristic extraction Download PDFInfo
- Publication number
- CN112653589A CN112653589A CN202010670869.2A CN202010670869A CN112653589A CN 112653589 A CN112653589 A CN 112653589A CN 202010670869 A CN202010670869 A CN 202010670869A CN 112653589 A CN112653589 A CN 112653589A
- Authority
- CN
- China
- Prior art keywords
- data flow
- deviation
- flow
- data
- host
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 18
- 238000000605 extraction Methods 0.000 title claims abstract description 15
- 230000005540 biological transmission Effects 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 12
- 230000002159 abnormal effect Effects 0.000 claims description 8
- 238000000034 method Methods 0.000 claims description 7
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 230000005856 abnormality Effects 0.000 claims description 4
- 238000012935 Averaging Methods 0.000 claims description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Environmental & Geological Engineering (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
A network data flow abnormity detection method based on host data flow characteristic extraction is characterized in that the characteristic extraction is carried out on the data flow of a host to be detected, the data flow is mapped to obtain a new characteristic collection with good separability, the new characteristic collection is put into a model by taking an average value to obtain a deviation threshold value, and finally, the network abnormity condition is judged according to the real-time network data flow deviation condition; the detection speed is high, and the accuracy is good.
Description
Technical Field
The invention relates to the field of network data flow anomaly detection, in particular to a network data flow anomaly detection method based on host data flow feature extraction.
Background
A network traffic anomaly refers to a situation where the behavior of network traffic deviates from its normal behavior. With the continuous enlargement of network scale and the continuous increase of complexity, the influence of network traffic abnormality on network performance is larger and larger. How to detect the abnormality of the network data flow quickly and accurately becomes a problem of great concern.
In order to solve the above problems, the present application provides a method for detecting network data flow anomaly based on host data flow feature extraction.
Disclosure of Invention
Objects of the invention
In order to solve the technical problems in the background art, the invention provides a network data flow abnormity detection method based on host data flow characteristic extraction, the invention extracts the characteristics of the data flow of a host to be detected, maps the data flow to obtain a new characteristic collection with good separability, obtains a deviation threshold value by taking an average value and introducing the average value into a model, and finally judges the network abnormity condition according to the real-time network data flow deviation condition; the detection speed is high, and the accuracy is good.
(II) technical scheme
In order to solve the above problems, the present invention provides a method for detecting network data flow abnormality based on host data flow feature extraction, which comprises the following steps:
s1, acquiring a historical flow data set of the host to be detected;
s2, preprocessing the historical flow data set; preprocessing the content, including missing value processing, format and content processing, and processing of removing repeated data and noise data;
s3, carrying out data flow feature extraction on the preprocessed historical flow data set: selecting and defining flow characteristic attributes in the historical flow data set from the n dimension, and mapping the defined characteristic attributes to the k dimension in a one-to-one correspondence manner to obtain new data flow characteristics;
s4, averaging the new data flow characteristics;
s5, establishing a deviation calculation model based on the average value in S4;
s6, substituting all new data flow characteristics into a deviation calculation model, and calculating the deviation of all new data flow characteristics;
s7, carrying out deviation summarization to obtain a deviation set, and setting a deviation threshold value;
s8, acquiring real-time data flow characteristics when flow abnormity detection is carried out;
s9, substituting the real-time data stream characteristics into a deviation calculation model to obtain real-time deviation;
and S10, judging the deviation condition of the real-time deviation according to the deviation threshold, judging that the network data flow is abnormal when the deviation condition exceeds the threshold, and judging that the network data flow is normal within the threshold range.
Preferably, the historical traffic data set in S1 includes the message transmission type, the message transmission number, the message transmission speed, and the message transmission content for all time windows in a plurality of specific time periods.
Preferably, in S2, the missing value processing requires removing fields, filling missing values, and re-fetching data according to the missing rate and importance.
Preferably, after the detection, the abnormal network data flow is stored in a database, and the deviation calculation model is optimized.
The technical scheme of the invention has the following beneficial technical effects:
the method comprises the steps of extracting features of a data stream of a host to be detected, mapping the data stream to obtain a new feature set with good separability, taking an average value, introducing the average value into a model to obtain a deviation threshold value, and finally judging a network abnormal condition according to a real-time network data stream deviation condition; the detection speed is high, and the accuracy is good.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the description is intended to be exemplary only, and is not intended to limit the scope of the present invention. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present invention.
The invention provides a network data flow abnormity detection method based on host data flow characteristic extraction, which comprises the following steps:
s1, acquiring a historical flow data set of the host to be detected;
s2, preprocessing the historical flow data set; preprocessing the content, including missing value processing, format and content processing, and processing of removing repeated data and noise data;
s3, carrying out data flow feature extraction on the preprocessed historical flow data set: selecting and defining flow characteristic attributes in the historical flow data set from the n dimension, and mapping the defined characteristic attributes to the k dimension in a one-to-one correspondence manner to obtain new data flow characteristics;
s4, averaging the new data flow characteristics;
s5, establishing a deviation calculation model based on the average value in S4;
s6, substituting all new data flow characteristics into a deviation calculation model, and calculating the deviation of all new data flow characteristics;
s7, carrying out deviation summarization to obtain a deviation set, and setting a deviation threshold value;
s8, acquiring real-time data flow characteristics when flow abnormity detection is carried out;
s9, substituting the real-time data stream characteristics into a deviation calculation model to obtain real-time deviation;
and S10, judging the deviation condition of the real-time deviation according to the deviation threshold, judging that the network data flow is abnormal when the deviation condition exceeds the threshold, and judging that the network data flow is normal within the threshold range.
In an alternative embodiment, the historical traffic data set in S1 includes the message transmission type, the message transmission number, the message transmission speed, and the message transmission content for all time windows in a plurality of specific time periods.
In an alternative embodiment, in S2, the missing value processing requires removing fields, filling missing values, and re-fetching data according to the missing rate and importance.
In an alternative embodiment, after detection, the abnormal network data stream is stored in a database, and the deviation calculation model is optimized.
In the invention, the data flow of a host to be detected is subjected to feature extraction and mapping to obtain a new feature set with good separability, and the new feature set is averaged and introduced into a model to obtain a deviation threshold value, and finally, the network abnormal condition is judged according to the real-time network data flow deviation condition; the detection speed is high, and the accuracy is good.
It is to be understood that the above-described embodiments of the present invention are merely illustrative of or explaining the principles of the invention and are not to be construed as limiting the invention. Therefore, any modification, equivalent replacement, improvement and the like made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. Further, it is intended that the appended claims cover all such variations and modifications as fall within the scope and boundaries of the appended claims or the equivalents of such scope and boundaries.
Claims (4)
1. A network data flow abnormity detection method based on host data flow characteristic extraction is characterized by comprising the following steps:
s1, acquiring a historical flow data set of the host to be detected;
s2, preprocessing the historical flow data set; preprocessing the content, including missing value processing, format and content processing, and processing of removing repeated data and noise data;
s3, carrying out data flow feature extraction on the preprocessed historical flow data set: selecting and defining flow characteristic attributes in the historical flow data set from the n dimension, and mapping the defined characteristic attributes to the k dimension in a one-to-one correspondence manner to obtain new data flow characteristics;
s4, averaging the new data flow characteristics;
s5, establishing a deviation calculation model based on the average value in S4;
s6, substituting all new data flow characteristics into a deviation calculation model, and calculating the deviation of all new data flow characteristics;
s7, carrying out deviation summarization to obtain a deviation set, and setting a deviation threshold value;
s8, acquiring real-time data flow characteristics when flow abnormity detection is carried out;
s9, substituting the real-time data stream characteristics into a deviation calculation model to obtain real-time deviation;
and S10, judging the deviation condition of the real-time deviation according to the deviation threshold, judging that the network data flow is abnormal when the deviation condition exceeds the threshold, and judging that the network data flow is normal within the threshold range.
2. The method according to claim 1, wherein the historical traffic data set in S1 includes message transmission types, message transmission numbers, message transmission speeds, and message transmission contents for all time windows in a plurality of specific time periods.
3. The method for detecting network data flow abnormality based on host data flow feature extraction as claimed in claim 1, wherein in S2, missing value processing requires removing fields, filling missing values, and re-fetching data according to missing rate and importance.
4. The method of claim 1, wherein after the detection, the abnormal network data flow is stored in a database, and the deviation calculation model is optimized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010670869.2A CN112653589A (en) | 2020-07-13 | 2020-07-13 | Network data flow abnormity detection method based on host data flow characteristic extraction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010670869.2A CN112653589A (en) | 2020-07-13 | 2020-07-13 | Network data flow abnormity detection method based on host data flow characteristic extraction |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112653589A true CN112653589A (en) | 2021-04-13 |
Family
ID=75346126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010670869.2A Pending CN112653589A (en) | 2020-07-13 | 2020-07-13 | Network data flow abnormity detection method based on host data flow characteristic extraction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112653589A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100290346A1 (en) * | 2006-11-29 | 2010-11-18 | Barford Paul R | Method and apparatus for network anomaly detection |
| CN102957579A (en) * | 2012-09-29 | 2013-03-06 | 北京邮电大学 | Network anomaly traffic monitoring method and device |
| CN104753733A (en) * | 2013-12-31 | 2015-07-01 | 中兴通讯股份有限公司 | Method and device for detecting abnormal network traffic data |
| CN106506242A (en) * | 2016-12-14 | 2017-03-15 | 北京东方棱镜科技有限公司 | A kind of Network anomalous behaviors and the accurate positioning method and system of flow monitoring |
| CN107483455A (en) * | 2017-08-25 | 2017-12-15 | 国家计算机网络与信息安全管理中心 | A kind of network node abnormality detection method and system based on stream |
| CN109120632A (en) * | 2018-09-04 | 2019-01-01 | 中国人民解放军陆军工程大学 | Network flow method for detecting abnormality based on online feature selection |
| CN111130932A (en) * | 2019-12-18 | 2020-05-08 | 北京浩瀚深度信息技术股份有限公司 | Method and device for predicting flow trend based on historical flow and storage medium |
| US20200177611A1 (en) * | 2017-12-06 | 2020-06-04 | Ribbon Communications Operating Company, Inc. | Communications methods and apparatus for dynamic detection and/or mitigation of anomalies |
-
2020
- 2020-07-13 CN CN202010670869.2A patent/CN112653589A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100290346A1 (en) * | 2006-11-29 | 2010-11-18 | Barford Paul R | Method and apparatus for network anomaly detection |
| CN102957579A (en) * | 2012-09-29 | 2013-03-06 | 北京邮电大学 | Network anomaly traffic monitoring method and device |
| CN104753733A (en) * | 2013-12-31 | 2015-07-01 | 中兴通讯股份有限公司 | Method and device for detecting abnormal network traffic data |
| CN106506242A (en) * | 2016-12-14 | 2017-03-15 | 北京东方棱镜科技有限公司 | A kind of Network anomalous behaviors and the accurate positioning method and system of flow monitoring |
| CN107483455A (en) * | 2017-08-25 | 2017-12-15 | 国家计算机网络与信息安全管理中心 | A kind of network node abnormality detection method and system based on stream |
| US20200177611A1 (en) * | 2017-12-06 | 2020-06-04 | Ribbon Communications Operating Company, Inc. | Communications methods and apparatus for dynamic detection and/or mitigation of anomalies |
| CN109120632A (en) * | 2018-09-04 | 2019-01-01 | 中国人民解放军陆军工程大学 | Network flow method for detecting abnormality based on online feature selection |
| CN111130932A (en) * | 2019-12-18 | 2020-05-08 | 北京浩瀚深度信息技术股份有限公司 | Method and device for predicting flow trend based on historical flow and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109729090B (en) | Slow denial of service attack detection method based on WEDMS clustering | |
| CN112788066B (en) | Abnormal flow detection method and system for Internet of things equipment and storage medium | |
| CN111131247B (en) | Vehicle-mounted internal network intrusion detection system | |
| CN112528277A (en) | Hybrid intrusion detection method based on recurrent neural network | |
| CN110602105B (en) | Large-scale parallelization network intrusion detection method based on k-means | |
| CN110661802A (en) | Low-speed denial of service attack detection method based on PCA-SVM algorithm | |
| CN111970229A (en) | CAN bus data anomaly detection method aiming at multiple attack modes | |
| CN115220396B (en) | Intelligent monitoring method and system for numerical control machine tool | |
| CN109951420B (en) | Multi-stage flow anomaly detection method based on entropy and dynamic linear relation | |
| CN114021135A (en) | LDoS attack detection and defense method based on R-SAX | |
| CN117148045A (en) | Fault studying and judging management system for running state of power distribution network | |
| CN110011966B (en) | Intelligent substation process layer network flow anomaly detection method | |
| CN116108402A (en) | Method, equipment and storage medium based on electric power multi-source heterogeneous data fusion analysis | |
| CN111191720A (en) | Service scene identification method and device and electronic equipment | |
| CN107070941A (en) | The method and apparatus of abnormal traffic detection | |
| CN112653589A (en) | Network data flow abnormity detection method based on host data flow characteristic extraction | |
| CN117072460A (en) | Centrifugal pump state monitoring method based on vibration data and expert experience | |
| CN117523299A (en) | Image recognition method, system and storage medium based on computer network | |
| CN105187451A (en) | Website flow abnormity detection method and system | |
| CN111865951A (en) | Network data flow abnormity detection method based on data packet feature extraction | |
| CN106295683A (en) | A kind of outlier detection method of time series data based on sharpness | |
| CN115295016A (en) | Equipment running state monitoring method, device, equipment and storage medium | |
| CN111025288B (en) | Security radar monitoring device and system | |
| CN112149579A (en) | Improved yolo _ v 3-based bird nest hidden danger detection method | |
| CN108650235B (en) | Intrusion detection device and detection method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220915 Address after: 361000 units 1702 and 1703, No. 59, Chengyi North Street, phase III, software park, Xiamen, Fujian Applicant after: XIAMEN USEEAR INFORMATION TECHNOLOGY Co.,Ltd. Address before: Unit 1701, unit 1704, No. 59, Chengyi North Street, phase III, software park, Xiamen City, Fujian Province, 361000 Applicant before: FUJIAN QIDIAN SPACE-TIME DIGITAL TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210413 |
|
| RJ01 | Rejection of invention patent application after publication |