CN105681303B - A kind of network safety situation monitoring of big data driving and method for visualizing - Google Patents

A kind of network safety situation monitoring of big data driving and method for visualizing Download PDF

Info

Publication number
CN105681303B
CN105681303B CN201610028522.1A CN201610028522A CN105681303B CN 105681303 B CN105681303 B CN 105681303B CN 201610028522 A CN201610028522 A CN 201610028522A CN 105681303 B CN105681303 B CN 105681303B
Authority
CN
China
Prior art keywords
attack
data
security
real time
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610028522.1A
Other languages
Chinese (zh)
Other versions
CN105681303A (en
Inventor
龙春
赵静
汪孔敏
于建军
万巍
高鹏
宋丹劼
王绍节
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Computer Network Information Center of CAS
Original Assignee
Computer Network Information Center of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Computer Network Information Center of CAS filed Critical Computer Network Information Center of CAS
Priority to CN201610028522.1A priority Critical patent/CN105681303B/en
Publication of CN105681303A publication Critical patent/CN105681303A/en
Application granted granted Critical
Publication of CN105681303B publication Critical patent/CN105681303B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/14Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1425Traffic logging, e.g. anomaly detection
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/14Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
    • H04L63/1416Event detection, e.g. attack signature detection
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00Network architectures or network communication protocols for network security
    • H04L63/14Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1433Vulnerability analysis

Abstract

Network safety situation monitoring and method for visualizing the present invention relates to a kind of driving of big data.This method comprises: 1) extract the basic network security data of different dimensions;2) basic network security data are stored and processed using Storm and Hadoop, wherein Hadoop is for handling historical data, and Strom is for handling real time data;3) Hadoop extracts crucial security feature item using big data processing method from historical data, and establishes database table structure, forms network security character knowledge base;4) Strom extracts relevant security feature item from real time data, it is carried out characteristic matching with network security character knowledge base, determines network safety situation;5) dynamic and visual displaying is carried out to the network safety situation that Strom determines.The present invention being capable of effectively monitoring network security postures, and the comprehensive visualization result for showing network safety situation.

Description

A kind of network safety situation monitoring of big data driving and method for visualizing
Technical field
The invention belongs to network technologies, field of information security technology, and in particular to a kind of network security of big data driving Situation monitoring and method for visualizing.
Background technique
The development of modern information technology has pushed generation, collection, transmission, the shared and analysis of data, so that science and work Journey research has become data-intensive work.Along with the increasingly increase of network flow, the type and complexity of attack Gradually promoted, be deployed in secure data provided by various security systems on network, equipment and platform have it is widely distributed, across The features such as tissue, format differences are big, magnanimity, nonumeric type, data dimension are promoted to multidimensional from single dimension, no matter from storage also It is to calculate aspect, the real-time accurate judgement of network safety situation cannot be all completed using traditional storage integration technology.
On the other hand, the data of higher-dimension magnanimity increase the work difficulty of Security Officer: (1) cognitive load is overweight, passes through Traditional log analysis mode analyze personnel be difficult within one day limited time to more than one hundred million alarm make in detail analysis and Judgement;(2) interactivity is inadequate, and when finding suspicious event, existing analysis mode can not provide related data filtering, event Details the functions such as shows to help to analyze personnel and make further effective judgement;(3) lack the understanding to the network overall situation, analysis What personnel often saw is all single data record, net that be difficult to identify that some complexity, that collaborative and the period is very long Network anomalous event.(4) log analysis based on traditional database is difficult to find some new attack modes, cannot be in advance to attack Trend make prediction or take precautions against in advance.
Summary of the invention
Based on above problem, the invention proposes a kind of network safety situation monitoring of real-time big data driving and visually Change method, being capable of effectively monitoring network security postures, and the comprehensive visualization result for showing network safety situation.
The technical solution adopted by the invention is as follows:
A kind of network safety situation monitoring of big data driving and method for visualizing, include the following steps:
1) the basic network security data of different dimensions, including real time data and historical data are extracted;
2) basic network security data are deposited using real time computation system Storm and distributed computing system Hadoop Storage and processing, wherein Hadoop is for handling historical data, and Strom is for handling real time data;
3) distributed computing system Hadoop extracts crucial safety using big data processing method from historical data Characteristic item, and database table structure is established, form network security character knowledge base;
4) real time computation system Strom extracts relevant security feature item from real time data, by itself and the network Security feature knowledge base carries out characteristic matching, and determines network safety situation according to matching result;
5) dynamic and visual displaying is carried out to the network safety situation that real time computation system Strom determines.
Further, the basic network security data of the step 1) different dimensions include website, host, longitude and latitude, IP Address, loophole, safety etc.;The historical data includes quarterly web site scan assessment report and quarterly host scanning assessment report It accuses.
Further, step 2) is for the treatment process of real time data: real time data will be sent to magnanimity day first Will paradigmatic system Flume, while data backup is carried out in HDFS system;The real time data being collected into is sent to point by Flume Cloth message system Kafka is to be further processed;By Kafka, treated that data flow is sent into real time computation system one by one Storm completes all real time business logics in Strom;Processing result is finally pressed into Redis in the form of similar stack to deposit Storage system, while web front end is extracted result and is shown from Redis.
Further, step 2) is for the treatment process of historical data: historical data being sent to pretreatment and integrates module After carrying out simple format analysis processing, it is sent to distributed computing system Hadoop and carries out big data analysis processing, then by statistics Simple data is deposited to Mysql database, non-structured data is stored to Hbase database, web front end no longer needs to be patrolled Processing is collected, the data directly read in database are shown.
Further, step 3) the big data processing method includes one of the following or a variety of: cluster and merging, closes Connection analysis, entropy analysis, Tendency Prediction;The security feature item includes: IP source address, IP destination address, event title, event class Not, security level, loophole code name.
Further, the step 5) visual presentation includes network security visualization and base based on network real-time traffic It is visualized in the network security of historical report, the content specifically visualized includes global dynamic attacks figure, domestic dynamic Attack graph, internal security situation map, national security breaches distribution map, announcement board and other function etc..
Beneficial effects of the present invention are as follows:
The present invention uses Hhadoop+Storm distributed structure/architecture to realize powerful bottom analysis ability;In order to complete Network safety situation is completely reflected in face, is extracted the different dimensions such as website, host, longitude and latitude, IP address, loophole, security incident Basic network security data;Real-time effective security postures evaluation result in order to obtain, establishes the security exception of self study Feature database;Show network safety situation as a result, using dynamic and visual technology in order to comprehensive.
The present invention by powerful bottom distributed storage and computation capability, each network flow of intelligent processing and Security log from various safety equipments obtains current most effective network safety situation judgment criteria, and in real time data Driving under carry out judging screening in real time, by result quick visualization, enabling safety analysis personnel in first time, monitoring is worked as The global safety situation, focal point risk public sentiment of preceding the whole network.
Detailed description of the invention
Fig. 1 is the Technical Architecture figure of overall plan of the present invention.
Fig. 2 is the disposed of in its entirety logic chart for the real-time data network security postures analysis that Storm is realized.
Fig. 3 is the process flow diagram of ReadBolt module.
Fig. 4 is the process flow diagram of IPBolt module.
Fig. 5 is the process flow diagram of RollCountBolt module.
Fig. 6 is the process flow diagram of FieldRankBolt module.
Fig. 7 is the process flow diagram of GlobalRankBolt module.
Fig. 8 is mass data visualization scheme schematic diagram.
Fig. 9 is that domestic dynamic attacks diagram is intended to.
Specific embodiment
Below by specific embodiments and the drawings, the present invention will be further described.
The network safety situation monitoring of real-time big data driving of the invention and method for visualizing, overall technology framework is such as Shown in Fig. 1, hadoop+storm distributed structure/architecture is generally used, to realize powerful bottom analysis ability.Of the invention Basic data source specifically includes that safety equipment real time data, quarterly web site scan assessment report (HTML), and quarterly host is swept Retouch assessment report (HTML).Wherein safety equipment real time data can use the IDS (Intrusion of network center Detection Systems intruding detection system) data etc..
As shown in Figure 1, collecting above-mentioned three kinds of data from the data transmit-receive server of hardware view, wherein real time data is (such as SYSLOG, i.e. system log) it will be sent to Flume, meanwhile, data backup is carried out on HDFS.Flume is received according to customized solution After collecting data, Kafka is sent it to, is further processed.Flume is a kind of massive logs paradigmatic system, is supported Various types of data sender is customized in system, for collecting data, meanwhile, it provides and simple process is carried out to data, and write various The ability of data receiver's (customizable).Kafka is a kind of distributed information system, in the website that can handle consumer's scale Everything flow data, solved by processing log and log aggregation according to the requirement of handling capacity.It is handled by Kafka Data flow afterwards will be sent into real time computation system Storm one by one, in Strom, complete all real time business logics, such as IP Address-organization information matching, geo-location, security incident classification of type statistics, high-risk research institute's information extraction etc..Finally, will These processing results are pressed into Redis storage system in the form of similar stack, meanwhile, it is aobvious that web front end extracts result from Redis Show.
And quarterly web site scan assessment report (HTML) and quarterly host scanning assessment report (HTML), i.e. in Fig. 1 Net.log and Server.log, will send to pretreatment and integrate after module carries out simple format analysis processing, be sent to distributed computing System Hadoop carries out big data analysis processing, and the process of processing includes cluster, association, statistics etc., in order to improve analysis result Storage efficiency, the simple data of statistics is deposited to Mysql database, and by non-structured data (i.e. non-relationship in Fig. 1 Type data) it will deposit to Hbase database, web front end no longer needs to carry out logical process, and the data directly read in database are shown ?.
Illustrate the detailed design of technical solution of the present invention below.
1. extraction, integration and the storage scheme of multidimensional network security postures data
The Chinese Academy of Sciences is used to extract the security log and net of interconnecting network equipment in each disposed a large amount of probes at present Network flow is formatted and is uniformly stored in traditional database or storage equipment.This mandatory formatization will lose some keys Information, also, in order to cater to traditional relationship type memory module, necessarily brought significantly to data query and analysis mechanisms Limitation.
Therefore, the characteristics of present invention is by analysis Various types of data, to retain most complete initial data as target, establishes multidimensional Model completes the mapping of peacekeeping measurement, obtains the multi-faceted data for being able to reflect network safety situation.And the storage of these data, Then use the multi-level storage of HDFS+Mysql+Hbase.
After real time data and flow access, by simple process and completes to sort out, be distributed to each recipient message queue In, wait the storage of next step to handle.Each message queue is managed with topic (session) for mark, is published to each The message of topic can be evenly dispersed to multiple partition (region), and when receiving subscription message, data flow is sent out Cloth is into real time computation system Storm, for the reliability of system, while carrying out HDFS storage.
After static data/historical data access, keyword extraction and pretreatment are carried out, is classified according to content/data format Storage forms the basic source database for being able to reflect network safety situation of a complete intelligence.
2. the network safety situation decision scheme of Real-time data drive
Real time data from bottom often has multiple types, such as network flow, device log, safety message etc., Often there is second the grade even delay of minute grade to the analysis processing of these data, so that security postures analysis result cannot be real When show, to reduce the validity of security monitoring.Therefore, for real-time exhibition demand, history and real-time linkage are proposed The solution of analysis is proposed to replace static data processing using the data analysis of linkage, be assisted using the experience of historical data Help the security decision of present real time data.
Based on above scheme, the skill combined using real time computation system Storm with distributed computing system Hadoop Art route.Wherein, distributed computing system Hadoop is mainly used for historical data (i.e. previously mentioned quarterly web site scan Assessment report and quarterly host scan assessment report) analysis, real time computation system Strom is then used for for real time data Processing and push.
Distributed computing system Hadoop integrates the multidimensional data collected from bottom, according to the historical data periodically inputted (i.e. previously mentioned quarterly web site scan assessment report and quarterly host scan assessment report), carries out data prediction, And using cluster with merge, association analysis, entropy analysis, the big datas processing method such as Tendency Prediction, from a large amount of, dynamic, mould Data correlation is found in the information security data of paste, learning network off-note forms network security character knowledge base.Such as Can extract crucial security feature item from data, including IP source address, IP destination address, event title, event category, Security level, loophole code name etc.;Later, database table structure is established, these security feature items are stored in multiple list items, shape At initial security feature library;The initial security feature library can regularly update.With endlessly data access, Hadoop Iterative analysis is constantly updated as a result, and finding new threat or being predicted.
Real time computation system Strom receiving real-time data, according to front-end business requirement extract key message item, and carrying out must The merger and discarding wanted.Meanwhile obscure with network security character knowledge base or accurately match, it, will if successful match Network safety situation evaluation result is pushed to front end in the form of WebSocket and is shown.
Specifically, after real time computation system Strom has handled a real time data, relevant security feature is extracted , characteristic matching is carried out with above-mentioned initial safe feature database, matched process is as follows:
1) according to IP destination address, if IP destination address is matched with the critical host ip in network security character knowledge base, Then defining the Host Security situation is crisis, is submitted to upper layer (front end visualization layer) as a result, and the host occur critical Number adds 1;
2) according to the security level of the data, if low, then directly filtering;If in, then according to security incident rule Which kind of classification number matching this security incident belongs to;If distributed denial of service attack (DDOS), then host peace is defined Total state is crisis, submits result to upper layer;It is attacked if detection scanning, then detection scanning counter adds 1, until reaching threshold value Just corresponding security incident type is considered as high-risk, submits result to upper layer;
3) network security character knowledge base maintenance one opens security incident-loophole and corresponds to table, and under normal circumstances, there are related leakages The host in hole encounters the attack of corresponding security incident, and maximum probability can generate serious consequence, therefore, when receiving a safe thing When part, the destination host that the security incident is searched in table is corresponded in security incident-loophole with the presence or absence of relevant vulnerability, and if it exists, It is then crisis state by the host definition, while does special mark in critical host ip table.
3. mass data visualization scheme
The program mainly solves how to show magnanimity high dimensional data in a manner of graph image, by people and number Image Communication is realized between, is enabled people to observe the mode implied in network security data, can quickly be found rule And find potential threaten.The mass data visualization scheme that the present invention realizes is broadly divided into the following contents:
1) the network security visualization based on network real-time traffic
Since the security incidents such as port scan, worm attack, Denial of Service attack have apparent a pair in terms of flow One, one-to-many or many-to-one feature, therefore, such attack often occurs in terms of flow apparent abnormal, shows net Network flow can help Network Safety Analysis personnel quickly to find network attack, preferably take precautions against and resist network intrusion event. Therefore, using the visualization display mode of point-to-point attack lines, display includes former IP address, purpose IP address, source port, mesh The information such as port, agreement, time, attack type.Meanwhile other methods are assisted while using the technology, such as use face Color mapping indicates different type attack etc..
2) the network security visualization based on historical report
In addition to real-time traffic, the historical report of magnanimity is also required to carry out the visualization of different dimensions according to different demands for security It shows.It include: security postures distribution/web portal security situation distribution of host in the whole country;Under different security domains, host Security postures distribution/web portal security situation distribution;IP address and security risk Distribution value relationship;Website and security risk Distribution value Relationship etc..
Specifically, global dynamic attacks figure, domestic dynamic attacks may be implemented in secure data Visualization Platform of the invention A variety of visual contents such as figure, internal security situation map, national security breaches distribution map, announcement board and other function, such as Fig. 8 institute Show.
A) function pages one: domestic dynamic attacks figure
Real-time attack in the whole country shows domestic each mechanism, the Chinese Academy of Sciences, mutually attacking between each research institute of the Chinese Academy of Sciences It hits, attack source 1, attack path 2 and target of attack 3 is shown on map of China, as shown in Figure 9.Simultaneously in four corners of the page (being not shown) show respectively attack source (Top10, by offensive attack number calculate) ranking list, target of attack (Top10, By offensive attack number calculate) ranking list, in real time attack information (include time, attack source, attack source IP, target of attack, target IP, attack type, attacked port etc.), attack type ranking list (Top10, by attack type calculate).
B) function pages two: global dynamic attacks figure
The real-time attack of global range shows attack of the global every country to 13 branch of the Chinese Academy of Sciences and its cooperation unit, Attack source, attack path and target of attack are shown on world map, page layout is same as above.
C) function pages three: internal security situation map
Page layout is divided into the part of left, center, right three, each three icons in two sides, and centre is map of China.In being above left side Highest eight research institutes of institute of section value-at-risk, centre are website vulnerability situation, lower section circulation display highest four nets of value-at-risk It stands;Circulation shows the security postures of each branch above right side, and centre is host loophole situation, lower section circulation display value-at-risk highest Four hosts;12 branches of the different display different colours on map of China according to value-at-risk mark.
D) function pages four: national security breaches distribution map
The page is divided into left and right two parts, is the security breaches situation scanned above left side, digital random increases to accordingly Loophole quantity, intercycle shows national loophole quantity ranking (Top8), Sql injection loophole ranking (Top8), cross site scripting The charts such as injection loophole ranking (Top8), the security breaches situation of lower section circulation display a research institute of the Chinese Academy of Sciences more than 100;Right side Circulation shows national security breaches, Sql injection loophole, cross site scripting injection loophole etc. on map of China.
E) function pages five: announcement board and other function
The page is divided into top, lower-left, bottom right three parts, and top is five icons, and click is switched to corresponding function The page, lower left circularly exhibiting bulletin, lower right show security bulletin and security breaches, tab bar switching can be used, the page is most Lower section shows copyright information.
4. (part Storm) is realized in the analysis design of real-time data network security postures
Distributed computing system Hadoop can be realized using the prior art in the present invention, therefore no longer be illustrated.4th Part mainly illustrates the specific implementation of real time computation system Storm.
In Storm, a figure shape structure for calculating in real time is first designed, referred to as topological (Topology).This A topology will be submitted to cluster, distribute code by the main controlled node (Master node) in cluster, assign the task to work Make node (Worker node) execution.It include Spout (data capsule) and the two kinds of angles Bolt (processing unit) in one topology Color, by Stream groupings (flow point group policy) by topology Spouts and Bolts connect, wherein Spout Message is sent, is responsible for sending data flow in the form of Tuple tuple;And Bolt is then responsible for converting these data flows, It can complete the operation such as to calculate, filter in Bolt, Bolt itself can also send the data to other Bolt at random.By Spout The Tuple launched is immutable array, corresponds to fixed key-value pair.
Present invention exploitation devises IPTopology (IP parsing topology), be by KafkaSpout, ReadBolt, These modules of IPBolt, RollCountBolt, FieldRankBolt and GlobalRankBolt pass through Shuffle Grouping (random grouping), Fields Grouping (being grouped by field) and Global Grouping (global packet) are even The topological diagram for picking up to be formed.IPTopology mainly completes the configuration of KafkaSpout, into topological diagram add Spouts and Bolts completes the tasks such as the configuration of operational mode.The KafkaSpout that the present invention uses Storm included is as data source. Kafka is a kind of distributed message processing facility.
4.1 disposed of in its entirety logics
The disposed of in its entirety logic for the real-time data network security postures analysis realized by Storm is as shown in Fig. 2, include such as Lower step, wherein KafkaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt and The specific implementation of these modules of GlobalRankBolt is described further below:
1.KafkaSpout constantly reads attack record from external data source, and attack record is transmitted to ReadBolt.
2.ReadBolt analyze attack record in attack source and attack purpose, then by attack source, target of attack with And original aggressor record is transmitted to IPBolt.
Responsible record to attack of 3.IPBolt carries out detailed analysis, and will attack Information encapsulation to SendData data knot Then Redis is written in structure;Meanwhile the information such as extracting attack source, target of attack, attack type are sent to RollCountBolt.
4.RollCountBolt mainly completes the statistics of the differential counting to attack source, target of attack, attack type, and will Statistical result is respectively stored in different Map structures (MAP in C Plus Plus), is then transmitted to all statistical results FieldRankBolt。
5. the different attack sources that each FieldRankBolt can receive oneself, different targets of attack, different attack classes All statistical results of type are ranked up, and screen respective topN list, and these sorted lists are transmitted to GlobalRankBolt。
6.GlobalRankBolt summarize the different attack sources that each FieldRankBolt sends, different targets of attack with And the sorted lists of different attack types, sequence is re-started, different attack sources, different targets of attack and different attacks are obtained Then the final topN list of type updates corresponding memory space in Redis.
4.2 IPTopology module designs
The KafkaSpout that the present invention uses Storm included is as data source.Kafka is a kind of distributed Message Processing Mechanism.One KafkaSpout can only handle the content of a topic, so including following information when initialization:
The address Broker (IP+Port) in 1.Kafka cluster;
2.topic name;
3. the unique identification Id of current Spout (following generation claims $ spout_id);
4. for storing currently processed offset Offset in distributed application program coordination service (zookeeper) (in following generation, claims $ zk_root);
5. data decoding process in current topic.
After initialization, need to add Spout and Bolt into Topology.It needs to be arranged before use when adding Spout Good configuration information kafkaConfig initializes Spout.When adding each Bolt, need using Stream Groupings (flow point group policy) defines each Bolt and receives which type of stream as input.Stream grouping defines one If a Stream should assign data to each task of each Bolt.Following three kinds of groupings have been used in the present embodiment respectively Type: random grouping;It is grouped by field;Global packet.After having added Spout and Bolt, match when Topology operation It sets.Configuring content includes run time behaviour, debugging enironment, cluster progress of work number etc..
4.3 KafkaSpout module designs
Spout is the message producer in a Topology.Under normal circumstances, the data source of Spout is outer from one Data are read from external data source in portion source, i.e. Spout, and to issuing message inside Topology: Tuple.
The KafkaSpout module in example that the present embodiment uses Storm included receives attack from outside as data source Then record emits data to Bolt.
4.4 ReadBolt module designs
ReadBolt module mainly extracts attack source IP address and attack mesh from the message that KafkaSpout transmitting comes IP address is marked, and processing result and original record are transmitted to IPBolt.Realizing the key method of ReadBolt module is:
1, calling interface, extracting attack record in attack source IP address and target of attack IP address.By attack source IP and Target of attack IP and original aggressor record are transmitted to IPBolt.
2, domain name and meaning that ReadBolt transmitting content includes are defined.ReadBolt emit content include " src ", " Dst ", " totally 3 domains line ", respectively correspond attack source IP, target of attack IP and original aggressor record.
The process flow of ReadBolt module is as shown in Figure 3, comprising: 1) calling interface, extracting attack record in attack Source IP address and target of attack IP address;2) attack source IP and target of attack IP and original aggressor record are transmitted to IPBolt。
4.5 IPBolt module designs
IPBolt module, which mainly has, completes following sections work:
1. the attack record character string that pair ReadBolt transmitting comes parses, extracting encapsulation SendData data structure is needed The each field wanted.The SendData data structure is for encapsulating attack graph (such as " global dynamic attacks figure ", " domestic dynamic Attack graph ") attack shown in the page records relevant each field, and these fields include: time, attack title, source IP address, source mechanism title, purpose IP address, purpose organization names, security incident type.
2. distinguishing extracting attack source IP, target of attack IP, and IpService (address of service IP library) is accessed, obtains attack Source IP, the unit of target of attack IP and location information.
3. the SendData.INSTANT_LOG_DATA of packaged SendData data structure write-in Redis is stored Space and SendData.INSTANT_OUR_DATA memory space, respectively " global dynamic attacks figure " and " domestic dynamic attacks Figure " provides basic data.
4. uniting respectively to provide initialization data when " global dynamic attacks figure ", the load of " the domestic dynamic attacks figure " page TopN (N=10 in this project) ranking of morning so far on the day of the meter whole world and domestic attack source, target of attack, attack type. So needing the information such as extracting attack source organization, target of attack organization, attack type title concurrent in IPBolt It penetrates to RollCountBolt, provides data for subsequent statistics ranking.
Shown in process flow Fig. 4 of IPBolt module, include the following steps:
1. the attack type number field first in parsing attack record character string obtains attack type number typeId.
2. judging whether to belong to the attack for ignoring type according to typeId.If it is the attack for ignoring type, then directly lose Abandon this attack record;It is no to then follow the steps 3.
3. lookup _ portNameMap obtains the corresponding attack type title typeName of typeId.If do not found, Attack type title typeName is assigned a value of " UNKNOWN TYPE ".
4. each field that remaining construction SendData data structure needs in parsing attack record character string.
5. the attack source IP, the target of attack IP that obtain from attack record are sent to IpService respectively, request IP's Unit and location information.
6. encapsulating Ip data structure according to the response message of IpService respectively, srcIp and outIp is constructed.
7. by srcIp, outIp, typeId, typeName and other fields extracted from attack record, envelope It is attached to SendData data structure, constructs data.
8. by the SendData.INSTANT_LOG_DATA of data write-in redis.
9. if the SendData.INSTANT_OUR_DATA of redis is written from the country in data by the srcIp of data.
10. extracting srcIp.Country, srcIp.City, outIp.City, typeName, logData, ourData hair It penetrates to RollCountBolt.
4.6 RollCountBolt module designs
The major function of RollCountBolt module is to complete to attack in global dynamic attacks and domestic dynamic attacks The counting in source, target of attack, attack type, and statistical result is transmitted to FieldRankBolt and carries out partial ordering.
The process flow of RollCountBolt module is as shown in figure 5, RollCountBolt receives an attack note every time When attack source, target of attack, attack type (be referred to as obj) information in record:
1.RollCountBolt first determines whether current time is morning, i.e., whether enters new one day.
1.1. if reaching morning, empty respectively global dynamic attacks and the corresponding attack source of domestic dynamic attacks, Target of attack, the Map structure of attack type (general designation _ objCounts) then execute step 2;
1.2. if not reaching morning, step 2 is directly executed.
2. judging whether this information belongs to domestic dynamic attacks.
2.1. if belonging to domestic dynamic attacks, respectively from global dynamic attacks and domestic dynamic attacks it is each _ The quantity of each obj of this information is obtained in objCounts;
2.2. if being not belonging to domestic dynamic attacks, only from each _ objCounts of global dynamic attacks obtain with This records the quantity of relevant each obj.
3. statistical magnitude is increased 1, and corresponding _ objCounts is updated using new value.
4. each obj and its corresponding statistical magnitude are transmitted to FieldRankBolt.
4.7 FieldRankBolt module designs
The function of FieldRankBolt module mainly completes some types in global dynamic attacks and domestic dynamic attacks Attack source, target of attack, attack type statistical information sequence, and periodic transmission is summarized to GlobalRankBolt. In one embodiment, transmit cycle can be 2s.
When receive global dynamic attacks, the attack source of domestic dynamic attacks, target of attack, attack type (being referred to as obj) and When its corresponding statistical magnitude, the process flow of FieldRankBolt module is as shown in fig. 6, include the following steps:
1. it is first determined whether needing to be arranged according to the ranking of this content update country each obj of dynamic attacks received Table.
1.1. if it is required, then to the corresponding ranked list of obj in global dynamic attacks, domestic dynamic attacks Reason, comprising: _ logSrcRank, _ logDstRank, _ logTypeRank, _ ourSrcRank, _ ourDstRank, _ ourTypeRank;
1.2. it if it is not required, then the corresponding ranked list of each obj is handled in detachment's whole world dynamic attacks, wraps It includes: _ logSrcRank, _ logDstRank, _ logTypeRank.
2. judging whether the content of this received each obj exists in corresponding ranked list.
2.1. if it is present directly updating corresponding value in corresponding lists using new value.
2.2. if it does not exist, then adding the obj received and its corresponding value in item list.
3. pair each ranked list reorders according to statistical magnitude descending.
4. deleting the sublist for exceeding topN in each ranked list.
5. each list, which is transmitted to GlobalRankBolt, carries out global sequence.
4.8 GlobalRankBolt module designs
The function of GlobalRankBolt module and the function of FieldRankBolt are similar, mainly each by summarizing The information that FieldRankBolt transmitting comes completes attack source, attack mesh in all global dynamic attacks and domestic dynamic attacks The sort method of mark, attack type, and regularly update corresponding memory space in Redis.In one embodiment, The update cycle of GlobalRankBolt can be 2s.
The process flow of GlobalRankBolt and the process flow of FieldRankBolt are consistent, as shown in fig. 7, but most The memory space of Redis can be directly updated afterwards.
The above embodiments are merely illustrative of the technical solutions of the present invention rather than is limited, the ordinary skill of this field Personnel can be with modification or equivalent replacement of the technical solution of the present invention are made, without departing from the spirit and scope of the present invention, this The protection scope of invention should be subject to described in claims.

Claims (8)

1. network safety situation monitoring and the method for visualizing of a kind of big data driving, which comprises the steps of:
1) the basic network security data of different dimensions, including real time data and historical data are extracted;
2) using real time computation system Storm and distributed computing system Hadoop to basic network security data carry out storage and Processing, wherein Hadoop is for handling historical data, and Storm is for handling real time data;
3) distributed computing system Hadoop extracts crucial security feature using big data processing method from historical data , and database table structure is established, form network security character knowledge base;
4) real time computation system Storm extracts relevant security feature item from real time data, by itself and the network security Feature knowledge library carries out characteristic matching, and determines network safety situation according to matching result;
5) dynamic and visual displaying is carried out to the network safety situation that real time computation system Storm determines;
Wherein, the step 4) method for carrying out characteristic matching is:
A) fixed if IP destination address is matched with the critical host ip in network security character knowledge base according to IP destination address The adopted Host Security situation is crisis, is submitted to upper layer, that is, front end visualization layer as a result, and critical number that the host occur Add 1;
B) according to the security level of the data, if low, then directly filtering;If in, then according to security incident rule numbers Match which kind of classification this security incident belongs to;If distributed denial of service attack, then the Host Security state is defined as danger Machine submits result to upper layer;It is attacked if detection scanning, then detection scanning counter adds 1, just will be corresponding until reaching threshold value Security incident type is considered as high-risk, submits result to upper layer;
C) the network security character knowledge base maintenance one opens security incident-loophole and corresponds to table, when receiving a security incident When, the destination host that the security incident is searched in table is corresponded in security incident-loophole with the presence or absence of relevant vulnerability, and if it exists, then It is crisis state by the host definition, while does special mark in critical host ip table;
Wherein, the step 5) visual presentation includes:
A) it the network security visualization based on network real-time traffic: using the visualization display mode of point-to-point attack lines, shows Show former IP address, purpose IP address, source port, destination port, agreement, time, attack type, while assisting other methods, wraps Including indicates different type attack using color mapping;
B) different dimensional the network security visualization based on historical report: is carried out according to different demands for security to the historical report of magnanimity The visual presentation of degree, comprising: in the whole country, the security postures distribution or the distribution of web portal security situation of host;Different safety Under domain, the security postures distribution or the distribution of web portal security situation of host;IP address and security risk Distribution value relationship;Website and peace Full blast danger Distribution value relationship.
2. the method as described in claim 1, which is characterized in that the basic network security data packet of the step 1) different dimensions Include website, host, longitude and latitude, IP address, loophole, security incident;The historical data includes that quarterly web site scan assessment is reported It accuses and quarterly host scans assessment report.
3. the method as described in claim 1, which is characterized in that step 2) is for the treatment process of real time data: first will Real time data will be sent to massive logs paradigmatic system Flume, while data backup is carried out in HDFS system;Flume will be received The real time data collected is sent to distributed information system Kafka to be further processed;By Kafka treated data flow It is sent into real time computation system Storm one by one, all real time business logics are completed in Storm;Finally by processing result with class It is pressed into Redis storage system like the form of stack, while web front end is extracted result and shown from Redis.
4. method as claimed in claim 3, which is characterized in that step 2) is for the treatment process of historical data: by history Data are sent to pretreatment and are integrated after module carries out simple format analysis processing, are sent to distributed computing system Hadoop and carry out big data Analysis processing, then the simple data of statistics is deposited to Mysql database, non-structured data are stored to Hbase data Library, web front end no longer need to carry out logical process, and the data directly read in database are shown.
5. the method as described in claim 1, which is characterized in that step 3) the big data processing method includes one in following Kind or it is a variety of: cluster with merge, association analysis, entropy analyze, Tendency Prediction;The security feature item includes: IP source address, IP mesh Mark address, event title, event category, security level, loophole code name.
6. the method as described in claim 1, which is characterized in that step 3) the network security character knowledge base regularly updates, As endlessly data access, Hadoop constantly update iterative analysis as a result, and finding new threat or being predicted.
7. the method as described in claim 1, which is characterized in that the content of the step 5) visual presentation includes: that the whole world is dynamic State attack graph, domestic dynamic attacks figure, internal security situation map, national security breaches distribution map, announcement board and other function.
8. the method as described in claim 1, which is characterized in that the real time computation system Storm includes following module: KafakaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt and GlobalRankBolt, this The disposed of in its entirety process of a little modules is as follows:
A) KafakaSpout constantly reads attack record from external data source, and attack record is transmitted to ReadBolt;
B) ReadBolt analyzes the attack source in attack record and attack purpose, then by attack source, target of attack and original Begin to attack to record and is transmitted to IPBolt;
C) IPBolt, which is responsible for recording attack, carries out detailed analysis, and by attack Information encapsulation to SendData data structure, so After Redis is written;The information such as extracting attack source, target of attack, attack type are sent to RollCountBolt simultaneously;
D) RollCountBolt mainly completes the statistics of the differential counting to attack source, target of attack, attack type, and will statistics As a result it is respectively stored in different Map structures, all statistical results is then transmitted to FieldRankBolt;
E) each FieldRankBolt receives oneself different attack sources, different targets of attack, different attack types institute There is statistical result to be ranked up, and screens respective topN list, and these sorted lists are transmitted to GlobalRankBolt;
F) GlobalRankBolt summarizes different attack sources, different targets of attack that each FieldRankBolt sends and not With the sorted lists of attack type, sequence is re-started, obtains different attack sources, different targets of attack and different attack types Then final topN list updates corresponding memory space in Redis.
CN201610028522.1A 2016-01-15 2016-01-15 A kind of network safety situation monitoring of big data driving and method for visualizing Active CN105681303B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610028522.1A CN105681303B (en) 2016-01-15 2016-01-15 A kind of network safety situation monitoring of big data driving and method for visualizing

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610028522.1A CN105681303B (en) 2016-01-15 2016-01-15 A kind of network safety situation monitoring of big data driving and method for visualizing

Publications (2)

Publication Number Publication Date
CN105681303A CN105681303A (en) 2016-06-15
CN105681303B true CN105681303B (en) 2019-02-01

Family

ID=56301035

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610028522.1A Active CN105681303B (en) 2016-01-15 2016-01-15 A kind of network safety situation monitoring of big data driving and method for visualizing

Country Status (1)

Country Link
CN (1) CN105681303B (en)

Families Citing this family (52)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106101252B (en) * 2016-07-01 2019-02-05 广西电网有限责任公司 Information Security Risk guard system based on big data and trust computing
CN106209856B (en) * 2016-07-14 2017-05-03 广西电网有限责任公司 Method for generating big data security posture map based on trusted computing
CN106254130B (en) * 2016-08-25 2019-06-07 华青融天(北京)技术股份有限公司 A kind of data processing method and device
CN106371986A (en) * 2016-09-08 2017-02-01 上海新炬网络技术有限公司 Log treatment operation and maintenance monitoring system
CN106407026A (en) * 2016-09-19 2017-02-15 北京集奥聚合科技有限公司 A method for generating message IDs in stream computing
CN106445790A (en) * 2016-10-12 2017-02-22 北京集奥聚合科技有限公司 Counting and account-checking method and device used in distributed real-time computing system
CN107943809B (en) * 2016-10-13 2022-02-01 阿里巴巴集团控股有限公司 Data quality monitoring method and device and big data computing platform
CN106599065B (en) * 2016-11-16 2019-12-13 北京化工大学 Food safety network public opinion early warning system based on Storm distributed framework
CN106682071A (en) * 2016-11-17 2017-05-17 安徽华博胜讯信息科技股份有限公司 University library digital resource sharing method based on big data
CN106528847A (en) * 2016-11-24 2017-03-22 北京集奥聚合科技有限公司 Multi-dimensional processing method and system for massive data
CN106874381B (en) * 2017-01-09 2020-12-22 重庆邮电大学 Radio environment map data processing system based on Hadoop
CN106850106B (en) * 2017-01-09 2020-05-12 重庆邮电大学 Radio environment map data collection system and method based on mobile crowd sensing
CN106941493B (en) * 2017-03-30 2020-02-18 北京奇艺世纪科技有限公司 Network security situation perception result output method and device
CN107169024A (en) * 2017-04-11 2017-09-15 微梦创科网络科技(中国)有限公司 The operation system and service implementation method of a kind of compatible type
CN107402997B (en) * 2017-07-20 2020-08-07 中国电子科技集团公司电子科学研究院 Security assessment method, terminal and computer storage medium for network public opinion situation
CN107483410A (en) * 2017-07-21 2017-12-15 中国联合网络通信集团有限公司 Network safety managing method and device
CN107508888A (en) * 2017-08-25 2017-12-22 同方(深圳)云计算技术股份有限公司 A kind of car networking service platform
CN107579855B (en) * 2017-09-21 2020-09-04 桂林电子科技大学 Layered multi-domain visual safe operation and maintenance method based on graph database
CN107786565A (en) * 2017-11-02 2018-03-09 江苏物联网研究发展中心 A kind of distributed real-time intrusion detection method and detecting system
CN108153828A (en) * 2017-12-12 2018-06-12 顺丰科技有限公司 A kind of persistence method of real time data, device and equipment, storage medium
CN108270785B (en) * 2018-01-15 2020-06-30 中国人民解放军国防科技大学 Knowledge graph-based distributed security event correlation analysis method
CN108197297B (en) * 2018-01-23 2020-09-29 正方软件股份有限公司 Data display method and system
CN108600300B (en) * 2018-03-06 2021-11-12 北京思空科技有限公司 Log data processing method and device
CN108242149B (en) * 2018-03-16 2020-06-30 成都智达万应科技有限公司 Big data analysis method based on traffic data
CN109088750B (en) * 2018-07-23 2021-05-25 下一代互联网重大应用技术(北京)工程研究中心有限公司 Container-based network situation awareness system design and deployment method
CN110881022A (en) * 2018-09-06 2020-03-13 福建雷盾信息安全有限公司 Large-scale network security situation detection and analysis method
CN109376325A (en) * 2018-09-26 2019-02-22 中国平安财产保险股份有限公司 User's institutional affiliation statistical method, device, computer equipment and storage medium
CN109598120A (en) * 2018-11-15 2019-04-09 中国科学院计算机网络信息中心 Security postures intelligent analysis method, device and the storage medium of mobile terminal
CN109299143B (en) * 2018-11-28 2022-03-22 重庆邮电大学 Knowledge fast indexing method of data interoperation test knowledge base based on Redis cache
CN109756381B (en) * 2019-02-11 2022-02-25 南方科技大学 Data center fault positioning method and device, electronic equipment and medium
CN110336785A (en) * 2019-05-22 2019-10-15 北京瀚海思创科技有限公司 The method for visualizing and storage medium of network attack chain figure
CN110213108A (en) * 2019-06-11 2019-09-06 四川久远国基科技有限公司 A kind of network security situation awareness method for early warning and system
CN110442550B (en) * 2019-07-05 2022-02-08 北京邮电大学 Log screen-gathering real-time visualization method and device
CN110554916B (en) * 2019-07-31 2022-07-29 苏宁云计算有限公司 Distributed cluster-based risk index calculation method and device
CN110460622B (en) * 2019-09-12 2021-11-16 贵州电网有限责任公司 Network anomaly detection method based on situation awareness prediction method
CN110716973A (en) * 2019-09-23 2020-01-21 杭州安恒信息技术股份有限公司 Big data based security event reporting platform and method
CN110764969A (en) * 2019-10-25 2020-02-07 新华三信息安全技术有限公司 Network attack tracing method and device
CN110855506A (en) * 2019-11-27 2020-02-28 国家电网有限公司信息通信分公司 Safety situation monitoring method and system
CN111193728B (en) * 2019-12-23 2022-04-01 成都烽创科技有限公司 Network security evaluation method, device, equipment and storage medium
CN111131253A (en) * 2019-12-24 2020-05-08 北京优特捷信息技术有限公司 Scene-based security event global response method, device, equipment and storage medium
CN111404879A (en) * 2020-02-26 2020-07-10 亚信科技(成都)有限公司 Visualization method and device for network threats
CN111562930A (en) * 2020-04-30 2020-08-21 深圳壹账通智能科技有限公司 Upgrading method and system for web application security
CN111935069B (en) * 2020-06-17 2022-08-26 西安理工大学 Traffic attack visualization characterization method based on time sequence
CN111787011B (en) * 2020-07-01 2022-03-29 公安部第三研究所 Intelligent analysis and early warning system, method and storage medium for security threat of information system
CN112532625B (en) * 2020-11-27 2022-09-13 杭州安恒信息安全技术有限公司 Network situation awareness evaluation data updating method and device and readable storage medium
CN112527879B (en) * 2020-12-15 2024-04-16 中国人寿保险股份有限公司 Kafka-based real-time data extraction method and related equipment
CN113438123B (en) * 2021-05-26 2022-08-30 曙光网络科技有限公司 Network flow monitoring method and device, computer equipment and storage medium
CN113596025A (en) * 2021-07-28 2021-11-02 中国南方电网有限责任公司 Power grid security event management method
CN115102778A (en) * 2022-07-11 2022-09-23 深信服科技股份有限公司 State determination method, device, equipment and medium
CN115643115B (en) * 2022-12-23 2023-03-10 武汉大学 Industrial control network security situation prediction method and system based on big data
CN117290413A (en) * 2023-08-05 2023-12-26 智参软件科技(上海)有限公司 Factory number real fusion platform based on SaaS and integration method
CN116756225B (en) * 2023-08-14 2023-11-07 南京展研信息技术有限公司 Situation data information processing method based on computer network security

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101547445A (en) * 2008-03-25 2009-09-30 上海摩波彼克半导体有限公司 System and method for detecting abnormal incursion based on mobility in mobile communication network
CN101883017A (en) * 2009-05-04 2010-11-10 北京启明星辰信息技术股份有限公司 System and method for evaluating network safe state
CN103345514A (en) * 2013-07-09 2013-10-09 焦点科技股份有限公司 Streamed data processing method in big data environment
US20130283233A1 (en) * 2012-04-24 2013-10-24 Maria Guadalupe Castellanos Multi-engine executable data-flow editor and translator
CN103593609A (en) * 2012-08-16 2014-02-19 阿里巴巴集团控股有限公司 Trustworthy behavior recognition method and device
CN104767757A (en) * 2015-04-17 2015-07-08 国家电网公司 Multiple-dimension security monitoring method and system based on WEB services

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101547445A (en) * 2008-03-25 2009-09-30 上海摩波彼克半导体有限公司 System and method for detecting abnormal incursion based on mobility in mobile communication network
CN101883017A (en) * 2009-05-04 2010-11-10 北京启明星辰信息技术股份有限公司 System and method for evaluating network safe state
US20130283233A1 (en) * 2012-04-24 2013-10-24 Maria Guadalupe Castellanos Multi-engine executable data-flow editor and translator
CN103593609A (en) * 2012-08-16 2014-02-19 阿里巴巴集团控股有限公司 Trustworthy behavior recognition method and device
CN103345514A (en) * 2013-07-09 2013-10-09 焦点科技股份有限公司 Streamed data processing method in big data environment
CN104767757A (en) * 2015-04-17 2015-07-08 国家电网公司 Multiple-dimension security monitoring method and system based on WEB services

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
"中国科技网网络安全平台及应用";宋丹劼,等;《科研信息化技术与应用》;20150630;全文 *
"基于Storm和Hadoop的大数据处理架构的研究";靳永超,等;《现代计算机(专业版)》;20150210;第0-2部分,图1 *
"网络安全分析中的大数据技术应用";王帅,等;《电信科学,2015年第07期》;20150731;第3-4部分,图1 *

Also Published As

Publication number Publication date
CN105681303A (en) 2016-06-15

Similar Documents

Publication Publication Date Title
CN105681303B (en) A kind of network safety situation monitoring of big data driving and method for visualizing
CN107196910B (en) Threat early warning monitoring system, method and deployment framework based on big data analysis
Aljawarneh et al. Anomaly-based intrusion detection system through feature selection analysis and building hybrid efficient model
CN106170772B (en) Network safety system
CN108270785A (en) Knowledge graph-based distributed security event correlation analysis method
Fischer et al. Real-time visual analytics for event data streams
Yadranjiaghdam et al. Developing a real-time data analytics framework for twitter streaming data
CN106487596A (en) Distributed Services follow the tracks of implementation method
CN102902813B (en) Result collection system
CN104767757A (en) Multiple-dimension security monitoring method and system based on WEB services
CN102111420A (en) Intelligent NIPS framework based on dynamic cloud/fire wall linkage
CN103927398A (en) Microblog hype group discovering method based on maximum frequent item set mining
CN109902297A (en) A kind of threat information generation method and device
CN108123939A (en) Malicious act real-time detection method and device
CN112765366A (en) APT (android Package) organization portrait construction method based on knowledge map
CN104516954A (en) Visualized evidence obtaining and analyzing system
Shi et al. Visual analytics of anomalous user behaviors: A survey
CN103886508A (en) Mass farmland data monitoring method and system
CN109710767A (en) Multilingual big data service platform
El Arass et al. Smart SIEM: From big data logs and events to smart data alerts
CN114430331A (en) Network security situation sensing method and system based on knowledge graph
Perrochon et al. Enlisting event patterns for cyber battlefield awareness
Campos et al. Creation and deployment of data mining-based intrusion detection systems in oracle database l0g
CN113938401A (en) Naval vessel network security visualization system
US20230065398A1 (en) Cygraph graph data ingest and enrichment pipeline

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant