CN110266603B - System and method for analyzing network flow of identity authentication service based on HTTP (hyper text transport protocol) - Google Patents
System and method for analyzing network flow of identity authentication service based on HTTP (hyper text transport protocol) Download PDFInfo
- Publication number
- CN110266603B CN110266603B CN201910569202.0A CN201910569202A CN110266603B CN 110266603 B CN110266603 B CN 110266603B CN 201910569202 A CN201910569202 A CN 201910569202A CN 110266603 B CN110266603 B CN 110266603B
- Authority
- CN
- China
- Prior art keywords
- network
- service node
- service
- flow
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0894—Packet rate
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/08—Network architectures or network communication protocols for network security for authentication of entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Abstract
The invention relates to a system and a method for analyzing network flow of identity authentication service based on HTTP protocol, wherein the system comprises an authentication platform, a communication interface, a memory, a back-end data analysis platform and flow acquisition equipment which are in communication connection; the network traffic collection device based on the HTTP protocol is deployed at the service node. The method comprises the steps of distinguishing parts needing to pay attention to network flow in the authentication platform in a service node mode through service flow, data flow and network planning of the authentication platform, deploying a hardware network flow collector, completing analysis, cleaning and storage of collected data through a Hadoop big data processing platform, and finally analyzing and early warning faults of the network based on the data. By using reasonable storage technology and structure, the number of data falling to the ground and the number of times of access are reduced, and the storage capacity and the storage efficiency of the authentication platform are improved.
Description
Technical Field
The invention relates to the technical field of computer software development programming, in particular to an identity authentication service network flow analysis system and method based on an HTTP (hyper text transport protocol).
Background
With the rapid development of internet technology, the real society extends to the depth of network space, and the network society becomes an important part of the real society. The internet brings the world-wide change to the life of people, and simultaneously obviously changes the behavior mode of people, and the behaviors and the services which are only generated in the real society before sending and receiving mails, social contact, shopping, bank transactions and the like quickly appear in a network space and are developed at a high speed.
In recent years, the technology of network authentication for identity of residents in China has gained a rapid pace, and the network scale is increasing exponentially with the number of users. Platforms and network architectures supporting the resident identity network authentication technology are also more and more complex, and the requirements on network fault handling are higher and higher. The method is a very practical and urgent problem of how to trace the circulation path of the data packet in the network, optimize the time for receiving and sending the data packet by the network node, and early warn the possible problems of the network node in advance. The method adopts a network flow analysis method based on an HTTP protocol, applies the traditional flow analysis method to identity authentication service, and has the advantages of acquiring inflow and outflow data of network flow of each service node in real time, analyzing the bottleneck of network equipment performance, mastering the characteristics of network operation, timely and properly adjusting the network, and ensuring normal and stable operation of the network.
(1) Acquisition interception and distributed storage of network messages
The network flow classification based on the machine learning method mainly classifies flows formed by network layer messages according to application types. A message is the smallest unit in a classification system. Dividing the Packets into two-way TCP or UDP flows according to the definition of the packet five-tuple (source IP address, source port number, target IP address, target port number and IP protocol), extracting the characteristics of the flows which are not related to the protocol and the ports to form a characteristic vector, and expressing the flows by the characteristic vector. There are many features characterizing network flows, and Andrew Moore lists 249 features of flows, and selects 37 candidate features with the best granularity as a feature set for classifying flows, specifically including the number and size features of the flows, the time features of the flows, and the flag bit information features of the flows. Based on the statistics and calculation of the characteristics, only the first 128 bytes of the message need to be intercepted. In system design, a WinPcap toolkit is adopted to capture a message, and the first 128 bytes of the message are intercepted to form a dmp file. And secondly, storing the message data to the HDFS in a distributed manner based on a Hadoop tool. The size of each block of the HDFS is 64M. In order to ensure that the Block of each HDFS can store the complete number of messages, part of information of the dmp file header is intercepted and stored in the NameNode of Hadoop in the test. Thus, the number of messages stored in each Block of the DataNode is 64 × 1024 × 1024/128=524 data message information.
(2) Formation of network flow based on MapReduce algorithm
The network flow forming process based on the MapReduce algorithm is divided into two stages. First, the Map phase is applied to the packet analysis of each block of the HDFS. According to the message information of 128 bytes, basic message data including time, capture length, protocol, source IP address and target IP address are extracted. Second, the Reduce phase is divided into multiple layers of reducers. The first layer Reducer is responsible for forming flow of the message information extracted in the Map stage according to the definition of the network flow and calculating the candidate characteristics of the flow; the second layer Reducer is responsible for merging the streams formed by the first layer Reducer according to the quintuple information.
The network flow analyzer has universality and has certain limitation on professional and directional analysis, and when a rear-end data analysis system is designed, a plurality of design data are consulted and referred to. At present, no network flow analysis method and system which are more suitable for an identity authentication platform are found. The following reasons are mainly used for the authentication platform:
1) The access amount and the concurrent access amount of the authentication platform are very large;
2) The communication protocols all adopt HTTP protocols;
3) The network structure of the authentication platform is complex;
4) The authentication platform adopts a multi-place and multi-center mode.
For these reasons, the network traffic analysis of the authentication platform is determined, and no existing network traffic analysis system is available.
Disclosure of Invention
In view of the defects of the prior art, the present invention aims to provide a system and a method for analyzing network traffic of identity authentication services based on the HTTP protocol, and a technical method for collecting and analyzing network traffic data of an authentication platform is achieved by using a single system.
Through service flow, data flow and network planning of the authentication platform, the part of the authentication platform which needs to pay attention to network flow is distinguished in a service node mode, then a hardware network flow collector is deployed, analysis, cleaning and storage of collected data are completed through a Hadoop big data processing platform, and finally analysis and fault early warning are carried out on the network based on the data.
By using reasonable storage technology and structure, the number of data falling to the ground and the number of times of access are reduced, and the storage capacity and the storage efficiency of the authentication platform are improved.
In order to solve the technical problems, the technical scheme provided by the invention is as follows:
an identity authentication service network flow analysis method based on HTTP protocol, the method includes the following steps:
s1, by combing and identifying service flow, data flow and network planning of an authentication platform, identifying a part, needing to pay attention to network flow, of the authentication platform in a service node mode;
s2, deploying network flow acquisition equipment based on an HTTP (hyper text transport protocol) protocol at a service node;
s3, mirroring the flow of each node to flow acquisition equipment, and acquiring the flow of each server node on each service link;
s4, analyzing, cleaning and storing the acquired data through a Hadoop big data processing platform;
s5, analyzing the network running state in real time according to the change condition of the network traffic data;
s6, discovering the bottleneck of the performance of the network equipment according to the inflow and outflow data of the network flow of the service node, grasping the characteristics of the network operation, and timely and properly adjusting the network to ensure the normal and stable operation of the network;
s7, setting a threshold value of service processing time of each node according to the processing performance of the service node, and judging the service performance condition of the service node by analyzing the processing time of the service node;
s8, setting a threshold value of the service transmission time of each node according to the service node transmission time, and judging the service condition of the service node by analyzing the transmission time of the service node.
It should be noted that, in the step 2, the amount of data flowing in and out of one service node can be obtained,
it should be noted that, in step 2, the sending time when one service node invokes another service node may be obtained, and the transmission time from one service node to another service node may be obtained.
It should be noted that the other service node receives the time and the other service node returns the start time of the one service node, and obtains the processing time of the other service node.
The identity authentication service network flow analysis system based on the HTTP comprises an authentication platform, a communication interface, a memory, a back-end data analysis platform and flow collection equipment, wherein the authentication platform is in communication connection with the communication interface; the network traffic collection device based on the HTTP protocol is deployed at the service node.
The invention has the beneficial effects that:
1) Early warning of network faults is more timely
The problem in the network is found according to the condition of the change of the network flow data, when the network flow approaches to 0, the network fault possibly occurs at the monitored position, and the problem needs to be processed in time.
2) The performance bottleneck in network planning can be timely discovered and adjusted
According to the analysis of the inflow data and the outflow data of the network flow, the bottleneck of the performance of the network equipment is found, the characteristics of the operation of the network are mastered, the network is adjusted appropriately in time, and the normal and stable operation of the network is ensured.
3) Accurately mastering the service performance of each service node in the network
And setting a threshold value of service processing time of each node according to the processing performance of the service node, and judging the service performance condition by analyzing the processing time of the service node.
Drawings
FIG. 1 is a schematic view of the present invention;
fig. 2 is a schematic diagram of a network traffic collection process according to the present invention.
Detailed Description
The invention is further illustrated by means of the following figures and examples, without thereby restricting the invention to the scope of the examples described.
As shown in fig. 1, the present invention is a method for analyzing network traffic of an identity authentication service based on an HTTP protocol, the method comprising the following steps:
s1, by combing and identifying service flow, data flow and network planning of an authentication platform, identifying a part, needing to pay attention to network flow, of the authentication platform in a service node mode;
s2, deploying network flow acquisition equipment based on an HTTP (hyper text transport protocol) protocol at a service node;
s3, mirroring the flow of each node to flow collection equipment, and acquiring the flow of each server node on each service link;
s4, analyzing, cleaning and storing the acquired data through a Hadoop big data processing platform;
s5, analyzing the running state of the network in real time according to the change condition of the network flow data;
s6, discovering the bottleneck of the performance of network equipment according to the inflow and outflow data of the network flow of the service node, grasping the characteristics of the network operation, and timely and properly adjusting the network to ensure the normal and stable operation of the network;
s7, setting a threshold value of service processing time of each node according to the processing performance of the service node, and judging the service performance condition of the service node by analyzing the processing time of the service node;
s8, setting a threshold value of the service transmission time of each node according to the service node transmission time, and judging the service condition of the service node by analyzing the transmission time of the service node.
It should be noted that, in the step 2, the incoming and outgoing data amount of one service node can be obtained,
it should be noted that, in step 2, the sending time when one service node invokes another service node may be obtained, and the transmission time from one service node to another service node may be obtained.
It should be noted that the other service node receives the time and the other service node returns the start time of the one service node, and obtains the processing time of the other service node.
As shown in fig. 2, the traffic of each node is mirrored to a traffic collection device F, the traffic of each server node on each service link is obtained, and when a service is called from a service node a to a service node B, the following contents are obtained through analysis of traffic data:
(1) The incoming and outgoing data volume of the service node B;
(2) The service node A calls the sending time of the service node B and the time of reaching the service node B, and the transmission time from the service node A to the service node B is obtained through calculation;
(2) And the B service node receives the time and the starting time when the B service node returns to the A service node, and the processing time of the B service is obtained through calculation.
From the data collected, the following analysis can also be obtained:
(1) Analyzing the running state of the network: finding out problems in the network according to the change condition of network flow data, and when the network flow approaches to 0, indicating that the monitored position may have network faults and needs to be processed in time;
(2) Analyzing network flow data with different time lengths: according to the inflow and outflow data of network flow, the bottleneck of the performance of network equipment is found, the characteristics of network operation are mastered, and the network is properly adjusted in time to ensure the normal and stable operation of the network;
(3) Analyzing the processing time of the service node: setting a threshold value of service processing time of each node according to the processing performance of the service node, and judging the service performance condition by analyzing the processing time of the service node;
(4) Analyzing the transmission time of the service node: and setting a threshold value of the service transmission time of each node according to the service node transmission time, and judging the service condition by analyzing the transmission time of the service node.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (2)
1. An identity authentication service network flow analysis method based on HTTP protocol is characterized by comprising the following steps:
s1, by combing and identifying the service flow, the data flow and the network plan of an authentication platform, identifying the part of the authentication platform which needs to pay attention to network flow in a service node manner;
s2, deploying network flow acquisition equipment based on an HTTP protocol at a service node;
s3, mirroring the flow of each service node to network flow acquisition equipment, and acquiring the flow of each service node on each service link;
s4, completing analysis, cleaning and storage of the acquired data through a HADOOP big data processing platform;
s5, analyzing the network running state in real time according to the change condition of the network flow data: finding out problems in the network according to the change condition of network flow data, and when the network flow approaches to 0, indicating that the monitored position may have network faults and needs to be processed in time;
s6, discovering the bottleneck of the performance of the network equipment according to the inflow and outflow data of the network flow of the service node, grasping the characteristics of the network operation, and timely and properly adjusting the network to ensure the normal and stable operation of the network;
s7, setting a threshold value of service processing time of each service node according to the processing performance of the service node, and judging the service performance condition of the service node by analyzing the processing time of the service node;
s8, setting a threshold value of service transmission time of each service node according to the service node transmission time, and judging the service condition of the service node by analyzing the transmission time of the service node;
the step S2 comprises the steps of obtaining the inflow and outflow data volume of one service node;
the step S2 includes obtaining a sending time when one service node calls another service node, obtaining a transmission time from one service node to another service node, obtaining a receiving time of another service node and a starting time when another service node returns to one service node, and obtaining a processing time of another service node.
2. A system for executing the HTTP protocol-based method for analyzing network traffic of identity authentication services according to claim 1, wherein the system includes an authentication platform, a communication interface, a memory, a backend data analysis platform, and a network traffic collection device, which are in communication connection with each other; the network traffic collection device based on the HTTP protocol is deployed at the service node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910569202.0A CN110266603B (en) | 2019-06-27 | 2019-06-27 | System and method for analyzing network flow of identity authentication service based on HTTP (hyper text transport protocol) |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910569202.0A CN110266603B (en) | 2019-06-27 | 2019-06-27 | System and method for analyzing network flow of identity authentication service based on HTTP (hyper text transport protocol) |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110266603A CN110266603A (en) | 2019-09-20 |
| CN110266603B true CN110266603B (en) | 2022-12-20 |
Family
ID=67922410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910569202.0A Active CN110266603B (en) | 2019-06-27 | 2019-06-27 | System and method for analyzing network flow of identity authentication service based on HTTP (hyper text transport protocol) |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110266603B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112069021B (en) * | 2020-08-21 | 2024-02-20 | 北京五八信息技术有限公司 | Flow data storage method and device, electronic equipment and storage medium |
| CN113542160A (en) * | 2021-05-27 | 2021-10-22 | 贵州电网有限责任公司 | SDN-based method and system for pulling east-west flow in cloud |
| CN114244582B (en) * | 2021-11-29 | 2023-06-20 | 国网江西省电力有限公司电力科学研究院 | Authentication method for low-profile data acquisition terminal associated with data in Internet of things |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104753732A (en) * | 2013-12-27 | 2015-07-01 | 郭祖龙 | Distribution based network traffic analysis system and method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7991827B1 (en) * | 2002-11-13 | 2011-08-02 | Mcafee, Inc. | Network analysis system and method utilizing collected metadata |
| CN102148716A (en) * | 2010-02-05 | 2011-08-10 | 中国联合网络通信集团有限公司 | Point-to-point system network performance testing method and system thereof |
| CN105610983A (en) * | 2016-03-07 | 2016-05-25 | 北京荣之联科技股份有限公司 | Distributive network monitoring method and system |
| CN105979532B (en) * | 2016-04-15 | 2020-01-03 | 北京思特奇信息技术股份有限公司 | Performance capacity analysis early warning method and device of service processing system |
| CN108833126B (en) * | 2018-04-02 | 2021-07-23 | 平安科技（深圳）有限公司 | Electronic device, data link risk early warning method and storage medium |
| CN109586999B (en) * | 2018-11-12 | 2021-03-23 | 深圳先进技术研究院 | Container cloud platform state monitoring and early warning system and method and electronic equipment |
-
2019
- 2019-06-27 CN CN201910569202.0A patent/CN110266603B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104753732A (en) * | 2013-12-27 | 2015-07-01 | 郭祖龙 | Distribution based network traffic analysis system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110266603A (en) | 2019-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111277578B (en) | Encrypted flow analysis feature extraction method, system, storage medium and security device | |
| CN106815112B (en) | Massive data monitoring system and method based on deep packet inspection | |
| CN110266603B (en) | System and method for analyzing network flow of identity authentication service based on HTTP (hyper text transport protocol) | |
| US9686157B2 (en) | Real-time adaptive processing of network data packets for analysis | |
| CN106921637A (en) | The recognition methods of the application message in network traffics and device | |
| CN102739457B (en) | Network flow recognition system and method based on DPI (Deep Packet Inspection) and SVM (Support Vector Machine) technology | |
| CN102035698B (en) | HTTP tunnel detection method based on decision tree classification algorithm | |
| US20050289231A1 (en) | System analysis program, system analysis method, and system analysis apparatus | |
| CN111935170A (en) | Network abnormal flow detection method, device and equipment | |
| CN109271793A (en) | Internet of Things cloud platform device class recognition methods and system | |
| CN111935063B (en) | Abnormal network access behavior monitoring system and method for terminal equipment | |
| EP3364627B1 (en) | Adaptive session intelligence extender | |
| CN110417729A (en) | A kind of service and application class method and system encrypting flow | |
| CN110855493A (en) | Application topological graph drawing device for mixed environment | |
| CN104486116A (en) | Multidimensional query method and multidimensional query system of flow data | |
| CN114281676A (en) | Black box fuzzy test method and system for industrial control private protocol | |
| CN101764754B (en) | Sample acquiring method in business identifying system based on DPI and DFI | |
| Cai et al. | Flow identification and characteristics mining from internet traffic with hadoop | |
| CN108512816A (en) | A kind of detection method and device that flow is kidnapped | |
| CN106446008A (en) | Management method and analysis system for database security event | |
| CN107360062B (en) | DPI equipment identification result verification method and system and DPI equipment | |
| KR20060063564A (en) | An apparatus for capturing internet protocol(ip) packet with sampling and signature searching function, and a method thereof | |
| CN112235154A (en) | Data processing method, system, device and medium based on Internet of things | |
| CN111917665A (en) | Terminal application data stream identification method and system | |
| CN113037551B (en) | Quick identification and positioning method for sensitive-related services based on traffic slice |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20210722 Address after: 100048 No. 1, South Road, capital gymnasium, Beijing, Haidian District Applicant after: THE FIRST Research Institute OF MINISTRY OF PUBLIC SECURITY Applicant after: Beijing ZHONGDUN Anxin Technology Development Co.,Ltd. Address before: 100048 No. 1, South Road, capital gymnasium, Beijing, Haidian District Applicant before: THE FIRST Research Institute OF MINISTRY OF PUBLIC SECURITY |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |