CN105681303B - A kind of network safety situation monitoring of big data driving and method for visualizing - Google Patents
A kind of network safety situation monitoring of big data driving and method for visualizing Download PDFInfo
- Publication number
- CN105681303B CN105681303B CN201610028522.1A CN201610028522A CN105681303B CN 105681303 B CN105681303 B CN 105681303B CN 201610028522 A CN201610028522 A CN 201610028522A CN 105681303 B CN105681303 B CN 105681303B
- Authority
- CN
- China
- Prior art keywords
- attack
- data
- security
- real time
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1433—Vulnerability analysis
Abstract
Network safety situation monitoring and method for visualizing the present invention relates to a kind of driving of big data.This method comprises: 1) extract the basic network security data of different dimensions;2) basic network security data are stored and processed using Storm and Hadoop, wherein Hadoop is for handling historical data, and Strom is for handling real time data;3) Hadoop extracts crucial security feature item using big data processing method from historical data, and establishes database table structure, forms network security character knowledge base;4) Strom extracts relevant security feature item from real time data, it is carried out characteristic matching with network security character knowledge base, determines network safety situation;5) dynamic and visual displaying is carried out to the network safety situation that Strom determines.The present invention being capable of effectively monitoring network security postures, and the comprehensive visualization result for showing network safety situation.
Description
Technical field
The invention belongs to network technologies, field of information security technology, and in particular to a kind of network security of big data driving
Situation monitoring and method for visualizing.
Background technique
The development of modern information technology has pushed generation, collection, transmission, the shared and analysis of data, so that science and work
Journey research has become data-intensive work.Along with the increasingly increase of network flow, the type and complexity of attack
Gradually promoted, be deployed in secure data provided by various security systems on network, equipment and platform have it is widely distributed, across
The features such as tissue, format differences are big, magnanimity, nonumeric type, data dimension are promoted to multidimensional from single dimension, no matter from storage also
It is to calculate aspect, the real-time accurate judgement of network safety situation cannot be all completed using traditional storage integration technology.
On the other hand, the data of higher-dimension magnanimity increase the work difficulty of Security Officer: (1) cognitive load is overweight, passes through
Traditional log analysis mode analyze personnel be difficult within one day limited time to more than one hundred million alarm make in detail analysis and
Judgement;(2) interactivity is inadequate, and when finding suspicious event, existing analysis mode can not provide related data filtering, event
Details the functions such as shows to help to analyze personnel and make further effective judgement;(3) lack the understanding to the network overall situation, analysis
What personnel often saw is all single data record, net that be difficult to identify that some complexity, that collaborative and the period is very long
Network anomalous event.(4) log analysis based on traditional database is difficult to find some new attack modes, cannot be in advance to attack
Trend make prediction or take precautions against in advance.
Summary of the invention
Based on above problem, the invention proposes a kind of network safety situation monitoring of real-time big data driving and visually
Change method, being capable of effectively monitoring network security postures, and the comprehensive visualization result for showing network safety situation.
The technical solution adopted by the invention is as follows:
A kind of network safety situation monitoring of big data driving and method for visualizing, include the following steps:
1) the basic network security data of different dimensions, including real time data and historical data are extracted;
2) basic network security data are deposited using real time computation system Storm and distributed computing system Hadoop
Storage and processing, wherein Hadoop is for handling historical data, and Strom is for handling real time data;
3) distributed computing system Hadoop extracts crucial safety using big data processing method from historical data
Characteristic item, and database table structure is established, form network security character knowledge base;
4) real time computation system Strom extracts relevant security feature item from real time data, by itself and the network
Security feature knowledge base carries out characteristic matching, and determines network safety situation according to matching result;
5) dynamic and visual displaying is carried out to the network safety situation that real time computation system Strom determines.
Further, the basic network security data of the step 1) different dimensions include website, host, longitude and latitude, IP
Address, loophole, safety etc.;The historical data includes quarterly web site scan assessment report and quarterly host scanning assessment report
It accuses.
Further, step 2) is for the treatment process of real time data: real time data will be sent to magnanimity day first
Will paradigmatic system Flume, while data backup is carried out in HDFS system;The real time data being collected into is sent to point by Flume
Cloth message system Kafka is to be further processed;By Kafka, treated that data flow is sent into real time computation system one by one
Storm completes all real time business logics in Strom;Processing result is finally pressed into Redis in the form of similar stack to deposit
Storage system, while web front end is extracted result and is shown from Redis.
Further, step 2) is for the treatment process of historical data: historical data being sent to pretreatment and integrates module
After carrying out simple format analysis processing, it is sent to distributed computing system Hadoop and carries out big data analysis processing, then by statistics
Simple data is deposited to Mysql database, non-structured data is stored to Hbase database, web front end no longer needs to be patrolled
Processing is collected, the data directly read in database are shown.
Further, step 3) the big data processing method includes one of the following or a variety of: cluster and merging, closes
Connection analysis, entropy analysis, Tendency Prediction;The security feature item includes: IP source address, IP destination address, event title, event class
Not, security level, loophole code name.
Further, the step 5) visual presentation includes network security visualization and base based on network real-time traffic
It is visualized in the network security of historical report, the content specifically visualized includes global dynamic attacks figure, domestic dynamic
Attack graph, internal security situation map, national security breaches distribution map, announcement board and other function etc..
Beneficial effects of the present invention are as follows:
The present invention uses Hhadoop+Storm distributed structure/architecture to realize powerful bottom analysis ability;In order to complete
Network safety situation is completely reflected in face, is extracted the different dimensions such as website, host, longitude and latitude, IP address, loophole, security incident
Basic network security data;Real-time effective security postures evaluation result in order to obtain, establishes the security exception of self study
Feature database;Show network safety situation as a result, using dynamic and visual technology in order to comprehensive.
The present invention by powerful bottom distributed storage and computation capability, each network flow of intelligent processing and
Security log from various safety equipments obtains current most effective network safety situation judgment criteria, and in real time data
Driving under carry out judging screening in real time, by result quick visualization, enabling safety analysis personnel in first time, monitoring is worked as
The global safety situation, focal point risk public sentiment of preceding the whole network.
Detailed description of the invention
Fig. 1 is the Technical Architecture figure of overall plan of the present invention.
Fig. 2 is the disposed of in its entirety logic chart for the real-time data network security postures analysis that Storm is realized.
Fig. 3 is the process flow diagram of ReadBolt module.
Fig. 4 is the process flow diagram of IPBolt module.
Fig. 5 is the process flow diagram of RollCountBolt module.
Fig. 6 is the process flow diagram of FieldRankBolt module.
Fig. 7 is the process flow diagram of GlobalRankBolt module.
Fig. 8 is mass data visualization scheme schematic diagram.
Fig. 9 is that domestic dynamic attacks diagram is intended to.
Specific embodiment
Below by specific embodiments and the drawings, the present invention will be further described.
The network safety situation monitoring of real-time big data driving of the invention and method for visualizing, overall technology framework is such as
Shown in Fig. 1, hadoop+storm distributed structure/architecture is generally used, to realize powerful bottom analysis ability.Of the invention
Basic data source specifically includes that safety equipment real time data, quarterly web site scan assessment report (HTML), and quarterly host is swept
Retouch assessment report (HTML).Wherein safety equipment real time data can use the IDS (Intrusion of network center
Detection Systems intruding detection system) data etc..
As shown in Figure 1, collecting above-mentioned three kinds of data from the data transmit-receive server of hardware view, wherein real time data is (such as
SYSLOG, i.e. system log) it will be sent to Flume, meanwhile, data backup is carried out on HDFS.Flume is received according to customized solution
After collecting data, Kafka is sent it to, is further processed.Flume is a kind of massive logs paradigmatic system, is supported
Various types of data sender is customized in system, for collecting data, meanwhile, it provides and simple process is carried out to data, and write various
The ability of data receiver's (customizable).Kafka is a kind of distributed information system, in the website that can handle consumer's scale
Everything flow data, solved by processing log and log aggregation according to the requirement of handling capacity.It is handled by Kafka
Data flow afterwards will be sent into real time computation system Storm one by one, in Strom, complete all real time business logics, such as IP
Address-organization information matching, geo-location, security incident classification of type statistics, high-risk research institute's information extraction etc..Finally, will
These processing results are pressed into Redis storage system in the form of similar stack, meanwhile, it is aobvious that web front end extracts result from Redis
Show.
And quarterly web site scan assessment report (HTML) and quarterly host scanning assessment report (HTML), i.e. in Fig. 1
Net.log and Server.log, will send to pretreatment and integrate after module carries out simple format analysis processing, be sent to distributed computing
System Hadoop carries out big data analysis processing, and the process of processing includes cluster, association, statistics etc., in order to improve analysis result
Storage efficiency, the simple data of statistics is deposited to Mysql database, and by non-structured data (i.e. non-relationship in Fig. 1
Type data) it will deposit to Hbase database, web front end no longer needs to carry out logical process, and the data directly read in database are shown
?.
Illustrate the detailed design of technical solution of the present invention below.
1. extraction, integration and the storage scheme of multidimensional network security postures data
The Chinese Academy of Sciences is used to extract the security log and net of interconnecting network equipment in each disposed a large amount of probes at present
Network flow is formatted and is uniformly stored in traditional database or storage equipment.This mandatory formatization will lose some keys
Information, also, in order to cater to traditional relationship type memory module, necessarily brought significantly to data query and analysis mechanisms
Limitation.
Therefore, the characteristics of present invention is by analysis Various types of data, to retain most complete initial data as target, establishes multidimensional
Model completes the mapping of peacekeeping measurement, obtains the multi-faceted data for being able to reflect network safety situation.And the storage of these data,
Then use the multi-level storage of HDFS+Mysql+Hbase.
After real time data and flow access, by simple process and completes to sort out, be distributed to each recipient message queue
In, wait the storage of next step to handle.Each message queue is managed with topic (session) for mark, is published to each
The message of topic can be evenly dispersed to multiple partition (region), and when receiving subscription message, data flow is sent out
Cloth is into real time computation system Storm, for the reliability of system, while carrying out HDFS storage.
After static data/historical data access, keyword extraction and pretreatment are carried out, is classified according to content/data format
Storage forms the basic source database for being able to reflect network safety situation of a complete intelligence.
2. the network safety situation decision scheme of Real-time data drive
Real time data from bottom often has multiple types, such as network flow, device log, safety message etc.,
Often there is second the grade even delay of minute grade to the analysis processing of these data, so that security postures analysis result cannot be real
When show, to reduce the validity of security monitoring.Therefore, for real-time exhibition demand, history and real-time linkage are proposed
The solution of analysis is proposed to replace static data processing using the data analysis of linkage, be assisted using the experience of historical data
Help the security decision of present real time data.
Based on above scheme, the skill combined using real time computation system Storm with distributed computing system Hadoop
Art route.Wherein, distributed computing system Hadoop is mainly used for historical data (i.e. previously mentioned quarterly web site scan
Assessment report and quarterly host scan assessment report) analysis, real time computation system Strom is then used for for real time data
Processing and push.
Distributed computing system Hadoop integrates the multidimensional data collected from bottom, according to the historical data periodically inputted
(i.e. previously mentioned quarterly web site scan assessment report and quarterly host scan assessment report), carries out data prediction,
And using cluster with merge, association analysis, entropy analysis, the big datas processing method such as Tendency Prediction, from a large amount of, dynamic, mould
Data correlation is found in the information security data of paste, learning network off-note forms network security character knowledge base.Such as
Can extract crucial security feature item from data, including IP source address, IP destination address, event title, event category,
Security level, loophole code name etc.;Later, database table structure is established, these security feature items are stored in multiple list items, shape
At initial security feature library;The initial security feature library can regularly update.With endlessly data access, Hadoop
Iterative analysis is constantly updated as a result, and finding new threat or being predicted.
Real time computation system Strom receiving real-time data, according to front-end business requirement extract key message item, and carrying out must
The merger and discarding wanted.Meanwhile obscure with network security character knowledge base or accurately match, it, will if successful match
Network safety situation evaluation result is pushed to front end in the form of WebSocket and is shown.
Specifically, after real time computation system Strom has handled a real time data, relevant security feature is extracted
, characteristic matching is carried out with above-mentioned initial safe feature database, matched process is as follows:
1) according to IP destination address, if IP destination address is matched with the critical host ip in network security character knowledge base,
Then defining the Host Security situation is crisis, is submitted to upper layer (front end visualization layer) as a result, and the host occur critical
Number adds 1;
2) according to the security level of the data, if low, then directly filtering;If in, then according to security incident rule
Which kind of classification number matching this security incident belongs to;If distributed denial of service attack (DDOS), then host peace is defined
Total state is crisis, submits result to upper layer;It is attacked if detection scanning, then detection scanning counter adds 1, until reaching threshold value
Just corresponding security incident type is considered as high-risk, submits result to upper layer;
3) network security character knowledge base maintenance one opens security incident-loophole and corresponds to table, and under normal circumstances, there are related leakages
The host in hole encounters the attack of corresponding security incident, and maximum probability can generate serious consequence, therefore, when receiving a safe thing
When part, the destination host that the security incident is searched in table is corresponded in security incident-loophole with the presence or absence of relevant vulnerability, and if it exists,
It is then crisis state by the host definition, while does special mark in critical host ip table.
3. mass data visualization scheme
The program mainly solves how to show magnanimity high dimensional data in a manner of graph image, by people and number
Image Communication is realized between, is enabled people to observe the mode implied in network security data, can quickly be found rule
And find potential threaten.The mass data visualization scheme that the present invention realizes is broadly divided into the following contents:
1) the network security visualization based on network real-time traffic
Since the security incidents such as port scan, worm attack, Denial of Service attack have apparent a pair in terms of flow
One, one-to-many or many-to-one feature, therefore, such attack often occurs in terms of flow apparent abnormal, shows net
Network flow can help Network Safety Analysis personnel quickly to find network attack, preferably take precautions against and resist network intrusion event.
Therefore, using the visualization display mode of point-to-point attack lines, display includes former IP address, purpose IP address, source port, mesh
The information such as port, agreement, time, attack type.Meanwhile other methods are assisted while using the technology, such as use face
Color mapping indicates different type attack etc..
2) the network security visualization based on historical report
In addition to real-time traffic, the historical report of magnanimity is also required to carry out the visualization of different dimensions according to different demands for security
It shows.It include: security postures distribution/web portal security situation distribution of host in the whole country;Under different security domains, host
Security postures distribution/web portal security situation distribution;IP address and security risk Distribution value relationship;Website and security risk Distribution value
Relationship etc..
Specifically, global dynamic attacks figure, domestic dynamic attacks may be implemented in secure data Visualization Platform of the invention
A variety of visual contents such as figure, internal security situation map, national security breaches distribution map, announcement board and other function, such as Fig. 8 institute
Show.
A) function pages one: domestic dynamic attacks figure
Real-time attack in the whole country shows domestic each mechanism, the Chinese Academy of Sciences, mutually attacking between each research institute of the Chinese Academy of Sciences
It hits, attack source 1, attack path 2 and target of attack 3 is shown on map of China, as shown in Figure 9.Simultaneously in four corners of the page
(being not shown) show respectively attack source (Top10, by offensive attack number calculate) ranking list, target of attack (Top10,
By offensive attack number calculate) ranking list, in real time attack information (include time, attack source, attack source IP, target of attack, target
IP, attack type, attacked port etc.), attack type ranking list (Top10, by attack type calculate).
B) function pages two: global dynamic attacks figure
The real-time attack of global range shows attack of the global every country to 13 branch of the Chinese Academy of Sciences and its cooperation unit,
Attack source, attack path and target of attack are shown on world map, page layout is same as above.
C) function pages three: internal security situation map
Page layout is divided into the part of left, center, right three, each three icons in two sides, and centre is map of China.In being above left side
Highest eight research institutes of institute of section value-at-risk, centre are website vulnerability situation, lower section circulation display highest four nets of value-at-risk
It stands;Circulation shows the security postures of each branch above right side, and centre is host loophole situation, lower section circulation display value-at-risk highest
Four hosts;12 branches of the different display different colours on map of China according to value-at-risk mark.
D) function pages four: national security breaches distribution map
The page is divided into left and right two parts, is the security breaches situation scanned above left side, digital random increases to accordingly
Loophole quantity, intercycle shows national loophole quantity ranking (Top8), Sql injection loophole ranking (Top8), cross site scripting
The charts such as injection loophole ranking (Top8), the security breaches situation of lower section circulation display a research institute of the Chinese Academy of Sciences more than 100;Right side
Circulation shows national security breaches, Sql injection loophole, cross site scripting injection loophole etc. on map of China.
E) function pages five: announcement board and other function
The page is divided into top, lower-left, bottom right three parts, and top is five icons, and click is switched to corresponding function
The page, lower left circularly exhibiting bulletin, lower right show security bulletin and security breaches, tab bar switching can be used, the page is most
Lower section shows copyright information.
4. (part Storm) is realized in the analysis design of real-time data network security postures
Distributed computing system Hadoop can be realized using the prior art in the present invention, therefore no longer be illustrated.4th
Part mainly illustrates the specific implementation of real time computation system Storm.
In Storm, a figure shape structure for calculating in real time is first designed, referred to as topological (Topology).This
A topology will be submitted to cluster, distribute code by the main controlled node (Master node) in cluster, assign the task to work
Make node (Worker node) execution.It include Spout (data capsule) and the two kinds of angles Bolt (processing unit) in one topology
Color, by Stream groupings (flow point group policy) by topology Spouts and Bolts connect, wherein Spout
Message is sent, is responsible for sending data flow in the form of Tuple tuple;And Bolt is then responsible for converting these data flows,
It can complete the operation such as to calculate, filter in Bolt, Bolt itself can also send the data to other Bolt at random.By Spout
The Tuple launched is immutable array, corresponds to fixed key-value pair.
Present invention exploitation devises IPTopology (IP parsing topology), be by KafkaSpout, ReadBolt,
These modules of IPBolt, RollCountBolt, FieldRankBolt and GlobalRankBolt pass through Shuffle
Grouping (random grouping), Fields Grouping (being grouped by field) and Global Grouping (global packet) are even
The topological diagram for picking up to be formed.IPTopology mainly completes the configuration of KafkaSpout, into topological diagram add Spouts and
Bolts completes the tasks such as the configuration of operational mode.The KafkaSpout that the present invention uses Storm included is as data source.
Kafka is a kind of distributed message processing facility.
4.1 disposed of in its entirety logics
The disposed of in its entirety logic for the real-time data network security postures analysis realized by Storm is as shown in Fig. 2, include such as
Lower step, wherein KafkaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt and
The specific implementation of these modules of GlobalRankBolt is described further below:
1.KafkaSpout constantly reads attack record from external data source, and attack record is transmitted to ReadBolt.
2.ReadBolt analyze attack record in attack source and attack purpose, then by attack source, target of attack with
And original aggressor record is transmitted to IPBolt.
Responsible record to attack of 3.IPBolt carries out detailed analysis, and will attack Information encapsulation to SendData data knot
Then Redis is written in structure;Meanwhile the information such as extracting attack source, target of attack, attack type are sent to RollCountBolt.
4.RollCountBolt mainly completes the statistics of the differential counting to attack source, target of attack, attack type, and will
Statistical result is respectively stored in different Map structures (MAP in C Plus Plus), is then transmitted to all statistical results
FieldRankBolt。
5. the different attack sources that each FieldRankBolt can receive oneself, different targets of attack, different attack classes
All statistical results of type are ranked up, and screen respective topN list, and these sorted lists are transmitted to
GlobalRankBolt。
6.GlobalRankBolt summarize the different attack sources that each FieldRankBolt sends, different targets of attack with
And the sorted lists of different attack types, sequence is re-started, different attack sources, different targets of attack and different attacks are obtained
Then the final topN list of type updates corresponding memory space in Redis.
4.2 IPTopology module designs
The KafkaSpout that the present invention uses Storm included is as data source.Kafka is a kind of distributed Message Processing
Mechanism.One KafkaSpout can only handle the content of a topic, so including following information when initialization:
The address Broker (IP+Port) in 1.Kafka cluster;
2.topic name;
3. the unique identification Id of current Spout (following generation claims $ spout_id);
4. for storing currently processed offset Offset in distributed application program coordination service (zookeeper)
(in following generation, claims $ zk_root);
5. data decoding process in current topic.
After initialization, need to add Spout and Bolt into Topology.It needs to be arranged before use when adding Spout
Good configuration information kafkaConfig initializes Spout.When adding each Bolt, need using Stream
Groupings (flow point group policy) defines each Bolt and receives which type of stream as input.Stream grouping defines one
If a Stream should assign data to each task of each Bolt.Following three kinds of groupings have been used in the present embodiment respectively
Type: random grouping;It is grouped by field;Global packet.After having added Spout and Bolt, match when Topology operation
It sets.Configuring content includes run time behaviour, debugging enironment, cluster progress of work number etc..
4.3 KafkaSpout module designs
Spout is the message producer in a Topology.Under normal circumstances, the data source of Spout is outer from one
Data are read from external data source in portion source, i.e. Spout, and to issuing message inside Topology: Tuple.
The KafkaSpout module in example that the present embodiment uses Storm included receives attack from outside as data source
Then record emits data to Bolt.
4.4 ReadBolt module designs
ReadBolt module mainly extracts attack source IP address and attack mesh from the message that KafkaSpout transmitting comes
IP address is marked, and processing result and original record are transmitted to IPBolt.Realizing the key method of ReadBolt module is:
1, calling interface, extracting attack record in attack source IP address and target of attack IP address.By attack source IP and
Target of attack IP and original aggressor record are transmitted to IPBolt.
2, domain name and meaning that ReadBolt transmitting content includes are defined.ReadBolt emit content include " src ", "
Dst ", " totally 3 domains line ", respectively correspond attack source IP, target of attack IP and original aggressor record.
The process flow of ReadBolt module is as shown in Figure 3, comprising: 1) calling interface, extracting attack record in attack
Source IP address and target of attack IP address;2) attack source IP and target of attack IP and original aggressor record are transmitted to
IPBolt。
4.5 IPBolt module designs
IPBolt module, which mainly has, completes following sections work:
1. the attack record character string that pair ReadBolt transmitting comes parses, extracting encapsulation SendData data structure is needed
The each field wanted.The SendData data structure is for encapsulating attack graph (such as " global dynamic attacks figure ", " domestic dynamic
Attack graph ") attack shown in the page records relevant each field, and these fields include: time, attack title, source
IP address, source mechanism title, purpose IP address, purpose organization names, security incident type.
2. distinguishing extracting attack source IP, target of attack IP, and IpService (address of service IP library) is accessed, obtains attack
Source IP, the unit of target of attack IP and location information.
3. the SendData.INSTANT_LOG_DATA of packaged SendData data structure write-in Redis is stored
Space and SendData.INSTANT_OUR_DATA memory space, respectively " global dynamic attacks figure " and " domestic dynamic attacks
Figure " provides basic data.
4. uniting respectively to provide initialization data when " global dynamic attacks figure ", the load of " the domestic dynamic attacks figure " page
TopN (N=10 in this project) ranking of morning so far on the day of the meter whole world and domestic attack source, target of attack, attack type.
So needing the information such as extracting attack source organization, target of attack organization, attack type title concurrent in IPBolt
It penetrates to RollCountBolt, provides data for subsequent statistics ranking.
Shown in process flow Fig. 4 of IPBolt module, include the following steps:
1. the attack type number field first in parsing attack record character string obtains attack type number typeId.
2. judging whether to belong to the attack for ignoring type according to typeId.If it is the attack for ignoring type, then directly lose
Abandon this attack record;It is no to then follow the steps 3.
3. lookup _ portNameMap obtains the corresponding attack type title typeName of typeId.If do not found,
Attack type title typeName is assigned a value of " UNKNOWN TYPE ".
4. each field that remaining construction SendData data structure needs in parsing attack record character string.
5. the attack source IP, the target of attack IP that obtain from attack record are sent to IpService respectively, request IP's
Unit and location information.
6. encapsulating Ip data structure according to the response message of IpService respectively, srcIp and outIp is constructed.
7. by srcIp, outIp, typeId, typeName and other fields extracted from attack record, envelope
It is attached to SendData data structure, constructs data.
8. by the SendData.INSTANT_LOG_DATA of data write-in redis.
9. if the SendData.INSTANT_OUR_DATA of redis is written from the country in data by the srcIp of data.
10. extracting srcIp.Country, srcIp.City, outIp.City, typeName, logData, ourData hair
It penetrates to RollCountBolt.
4.6 RollCountBolt module designs
The major function of RollCountBolt module is to complete to attack in global dynamic attacks and domestic dynamic attacks
The counting in source, target of attack, attack type, and statistical result is transmitted to FieldRankBolt and carries out partial ordering.
The process flow of RollCountBolt module is as shown in figure 5, RollCountBolt receives an attack note every time
When attack source, target of attack, attack type (be referred to as obj) information in record:
1.RollCountBolt first determines whether current time is morning, i.e., whether enters new one day.
1.1. if reaching morning, empty respectively global dynamic attacks and the corresponding attack source of domestic dynamic attacks,
Target of attack, the Map structure of attack type (general designation _ objCounts) then execute step 2;
1.2. if not reaching morning, step 2 is directly executed.
2. judging whether this information belongs to domestic dynamic attacks.
2.1. if belonging to domestic dynamic attacks, respectively from global dynamic attacks and domestic dynamic attacks it is each _
The quantity of each obj of this information is obtained in objCounts;
2.2. if being not belonging to domestic dynamic attacks, only from each _ objCounts of global dynamic attacks obtain with
This records the quantity of relevant each obj.
3. statistical magnitude is increased 1, and corresponding _ objCounts is updated using new value.
4. each obj and its corresponding statistical magnitude are transmitted to FieldRankBolt.
4.7 FieldRankBolt module designs
The function of FieldRankBolt module mainly completes some types in global dynamic attacks and domestic dynamic attacks
Attack source, target of attack, attack type statistical information sequence, and periodic transmission is summarized to GlobalRankBolt.
In one embodiment, transmit cycle can be 2s.
When receive global dynamic attacks, the attack source of domestic dynamic attacks, target of attack, attack type (being referred to as obj) and
When its corresponding statistical magnitude, the process flow of FieldRankBolt module is as shown in fig. 6, include the following steps:
1. it is first determined whether needing to be arranged according to the ranking of this content update country each obj of dynamic attacks received
Table.
1.1. if it is required, then to the corresponding ranked list of obj in global dynamic attacks, domestic dynamic attacks
Reason, comprising: _ logSrcRank, _ logDstRank, _ logTypeRank, _ ourSrcRank, _ ourDstRank, _
ourTypeRank;
1.2. it if it is not required, then the corresponding ranked list of each obj is handled in detachment's whole world dynamic attacks, wraps
It includes: _ logSrcRank, _ logDstRank, _ logTypeRank.
2. judging whether the content of this received each obj exists in corresponding ranked list.
2.1. if it is present directly updating corresponding value in corresponding lists using new value.
2.2. if it does not exist, then adding the obj received and its corresponding value in item list.
3. pair each ranked list reorders according to statistical magnitude descending.
4. deleting the sublist for exceeding topN in each ranked list.
5. each list, which is transmitted to GlobalRankBolt, carries out global sequence.
4.8 GlobalRankBolt module designs
The function of GlobalRankBolt module and the function of FieldRankBolt are similar, mainly each by summarizing
The information that FieldRankBolt transmitting comes completes attack source, attack mesh in all global dynamic attacks and domestic dynamic attacks
The sort method of mark, attack type, and regularly update corresponding memory space in Redis.In one embodiment,
The update cycle of GlobalRankBolt can be 2s.
The process flow of GlobalRankBolt and the process flow of FieldRankBolt are consistent, as shown in fig. 7, but most
The memory space of Redis can be directly updated afterwards.
The above embodiments are merely illustrative of the technical solutions of the present invention rather than is limited, the ordinary skill of this field
Personnel can be with modification or equivalent replacement of the technical solution of the present invention are made, without departing from the spirit and scope of the present invention, this
The protection scope of invention should be subject to described in claims.
Claims (8)
1. network safety situation monitoring and the method for visualizing of a kind of big data driving, which comprises the steps of:
1) the basic network security data of different dimensions, including real time data and historical data are extracted;
2) using real time computation system Storm and distributed computing system Hadoop to basic network security data carry out storage and
Processing, wherein Hadoop is for handling historical data, and Storm is for handling real time data;
3) distributed computing system Hadoop extracts crucial security feature using big data processing method from historical data
, and database table structure is established, form network security character knowledge base;
4) real time computation system Storm extracts relevant security feature item from real time data, by itself and the network security
Feature knowledge library carries out characteristic matching, and determines network safety situation according to matching result;
5) dynamic and visual displaying is carried out to the network safety situation that real time computation system Storm determines;
Wherein, the step 4) method for carrying out characteristic matching is:
A) fixed if IP destination address is matched with the critical host ip in network security character knowledge base according to IP destination address
The adopted Host Security situation is crisis, is submitted to upper layer, that is, front end visualization layer as a result, and critical number that the host occur
Add 1;
B) according to the security level of the data, if low, then directly filtering;If in, then according to security incident rule numbers
Match which kind of classification this security incident belongs to;If distributed denial of service attack, then the Host Security state is defined as danger
Machine submits result to upper layer;It is attacked if detection scanning, then detection scanning counter adds 1, just will be corresponding until reaching threshold value
Security incident type is considered as high-risk, submits result to upper layer;
C) the network security character knowledge base maintenance one opens security incident-loophole and corresponds to table, when receiving a security incident
When, the destination host that the security incident is searched in table is corresponded in security incident-loophole with the presence or absence of relevant vulnerability, and if it exists, then
It is crisis state by the host definition, while does special mark in critical host ip table;
Wherein, the step 5) visual presentation includes:
A) it the network security visualization based on network real-time traffic: using the visualization display mode of point-to-point attack lines, shows
Show former IP address, purpose IP address, source port, destination port, agreement, time, attack type, while assisting other methods, wraps
Including indicates different type attack using color mapping;
B) different dimensional the network security visualization based on historical report: is carried out according to different demands for security to the historical report of magnanimity
The visual presentation of degree, comprising: in the whole country, the security postures distribution or the distribution of web portal security situation of host;Different safety
Under domain, the security postures distribution or the distribution of web portal security situation of host;IP address and security risk Distribution value relationship;Website and peace
Full blast danger Distribution value relationship.
2. the method as described in claim 1, which is characterized in that the basic network security data packet of the step 1) different dimensions
Include website, host, longitude and latitude, IP address, loophole, security incident;The historical data includes that quarterly web site scan assessment is reported
It accuses and quarterly host scans assessment report.
3. the method as described in claim 1, which is characterized in that step 2) is for the treatment process of real time data: first will
Real time data will be sent to massive logs paradigmatic system Flume, while data backup is carried out in HDFS system;Flume will be received
The real time data collected is sent to distributed information system Kafka to be further processed;By Kafka treated data flow
It is sent into real time computation system Storm one by one, all real time business logics are completed in Storm;Finally by processing result with class
It is pressed into Redis storage system like the form of stack, while web front end is extracted result and shown from Redis.
4. method as claimed in claim 3, which is characterized in that step 2) is for the treatment process of historical data: by history
Data are sent to pretreatment and are integrated after module carries out simple format analysis processing, are sent to distributed computing system Hadoop and carry out big data
Analysis processing, then the simple data of statistics is deposited to Mysql database, non-structured data are stored to Hbase data
Library, web front end no longer need to carry out logical process, and the data directly read in database are shown.
5. the method as described in claim 1, which is characterized in that step 3) the big data processing method includes one in following
Kind or it is a variety of: cluster with merge, association analysis, entropy analyze, Tendency Prediction;The security feature item includes: IP source address, IP mesh
Mark address, event title, event category, security level, loophole code name.
6. the method as described in claim 1, which is characterized in that step 3) the network security character knowledge base regularly updates,
As endlessly data access, Hadoop constantly update iterative analysis as a result, and finding new threat or being predicted.
7. the method as described in claim 1, which is characterized in that the content of the step 5) visual presentation includes: that the whole world is dynamic
State attack graph, domestic dynamic attacks figure, internal security situation map, national security breaches distribution map, announcement board and other function.
8. the method as described in claim 1, which is characterized in that the real time computation system Storm includes following module:
KafakaSpout, ReadBolt, IPBolt, RollCountBolt, FieldRankBolt and GlobalRankBolt, this
The disposed of in its entirety process of a little modules is as follows:
A) KafakaSpout constantly reads attack record from external data source, and attack record is transmitted to ReadBolt;
B) ReadBolt analyzes the attack source in attack record and attack purpose, then by attack source, target of attack and original
Begin to attack to record and is transmitted to IPBolt;
C) IPBolt, which is responsible for recording attack, carries out detailed analysis, and by attack Information encapsulation to SendData data structure, so
After Redis is written;The information such as extracting attack source, target of attack, attack type are sent to RollCountBolt simultaneously;
D) RollCountBolt mainly completes the statistics of the differential counting to attack source, target of attack, attack type, and will statistics
As a result it is respectively stored in different Map structures, all statistical results is then transmitted to FieldRankBolt;
E) each FieldRankBolt receives oneself different attack sources, different targets of attack, different attack types institute
There is statistical result to be ranked up, and screens respective topN list, and these sorted lists are transmitted to GlobalRankBolt;
F) GlobalRankBolt summarizes different attack sources, different targets of attack that each FieldRankBolt sends and not
With the sorted lists of attack type, sequence is re-started, obtains different attack sources, different targets of attack and different attack types
Then final topN list updates corresponding memory space in Redis.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610028522.1A CN105681303B (en) | 2016-01-15 | 2016-01-15 | A kind of network safety situation monitoring of big data driving and method for visualizing |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610028522.1A CN105681303B (en) | 2016-01-15 | 2016-01-15 | A kind of network safety situation monitoring of big data driving and method for visualizing |
Publications (2)
Publication Number | Publication Date |
---|---|
CN105681303A CN105681303A (en) | 2016-06-15 |
CN105681303B true CN105681303B (en) | 2019-02-01 |
Family
ID=56301035
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610028522.1A Active CN105681303B (en) | 2016-01-15 | 2016-01-15 | A kind of network safety situation monitoring of big data driving and method for visualizing |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105681303B (en) |
Families Citing this family (52)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106101252B (en) * | 2016-07-01 | 2019-02-05 | 广西电网有限责任公司 | Information Security Risk guard system based on big data and trust computing |
CN106209856B (en) * | 2016-07-14 | 2017-05-03 | 广西电网有限责任公司 | Method for generating big data security posture map based on trusted computing |
CN106254130B (en) * | 2016-08-25 | 2019-06-07 | 华青融天(北京)技术股份有限公司 | A kind of data processing method and device |
CN106371986A (en) * | 2016-09-08 | 2017-02-01 | 上海新炬网络技术有限公司 | Log treatment operation and maintenance monitoring system |
CN106407026A (en) * | 2016-09-19 | 2017-02-15 | 北京集奥聚合科技有限公司 | A method for generating message IDs in stream computing |
CN106445790A (en) * | 2016-10-12 | 2017-02-22 | 北京集奥聚合科技有限公司 | Counting and account-checking method and device used in distributed real-time computing system |
CN107943809B (en) * | 2016-10-13 | 2022-02-01 | 阿里巴巴集团控股有限公司 | Data quality monitoring method and device and big data computing platform |
CN106599065B (en) * | 2016-11-16 | 2019-12-13 | 北京化工大学 | Food safety network public opinion early warning system based on Storm distributed framework |
CN106682071A (en) * | 2016-11-17 | 2017-05-17 | 安徽华博胜讯信息科技股份有限公司 | University library digital resource sharing method based on big data |
CN106528847A (en) * | 2016-11-24 | 2017-03-22 | 北京集奥聚合科技有限公司 | Multi-dimensional processing method and system for massive data |
CN106874381B (en) * | 2017-01-09 | 2020-12-22 | 重庆邮电大学 | Radio environment map data processing system based on Hadoop |
CN106850106B (en) * | 2017-01-09 | 2020-05-12 | 重庆邮电大学 | Radio environment map data collection system and method based on mobile crowd sensing |
CN106941493B (en) * | 2017-03-30 | 2020-02-18 | 北京奇艺世纪科技有限公司 | Network security situation perception result output method and device |
CN107169024A (en) * | 2017-04-11 | 2017-09-15 | 微梦创科网络科技(中国)有限公司 | The operation system and service implementation method of a kind of compatible type |
CN107402997B (en) * | 2017-07-20 | 2020-08-07 | 中国电子科技集团公司电子科学研究院 | Security assessment method, terminal and computer storage medium for network public opinion situation |
CN107483410A (en) * | 2017-07-21 | 2017-12-15 | 中国联合网络通信集团有限公司 | Network safety managing method and device |
CN107508888A (en) * | 2017-08-25 | 2017-12-22 | 同方(深圳)云计算技术股份有限公司 | A kind of car networking service platform |
CN107579855B (en) * | 2017-09-21 | 2020-09-04 | 桂林电子科技大学 | Layered multi-domain visual safe operation and maintenance method based on graph database |
CN107786565A (en) * | 2017-11-02 | 2018-03-09 | 江苏物联网研究发展中心 | A kind of distributed real-time intrusion detection method and detecting system |
CN108153828A (en) * | 2017-12-12 | 2018-06-12 | 顺丰科技有限公司 | A kind of persistence method of real time data, device and equipment, storage medium |
CN108270785B (en) * | 2018-01-15 | 2020-06-30 | 中国人民解放军国防科技大学 | Knowledge graph-based distributed security event correlation analysis method |
CN108197297B (en) * | 2018-01-23 | 2020-09-29 | 正方软件股份有限公司 | Data display method and system |
CN108600300B (en) * | 2018-03-06 | 2021-11-12 | 北京思空科技有限公司 | Log data processing method and device |
CN108242149B (en) * | 2018-03-16 | 2020-06-30 | 成都智达万应科技有限公司 | Big data analysis method based on traffic data |
CN109088750B (en) * | 2018-07-23 | 2021-05-25 | 下一代互联网重大应用技术(北京)工程研究中心有限公司 | Container-based network situation awareness system design and deployment method |
CN110881022A (en) * | 2018-09-06 | 2020-03-13 | 福建雷盾信息安全有限公司 | Large-scale network security situation detection and analysis method |
CN109376325A (en) * | 2018-09-26 | 2019-02-22 | 中国平安财产保险股份有限公司 | User's institutional affiliation statistical method, device, computer equipment and storage medium |
CN109598120A (en) * | 2018-11-15 | 2019-04-09 | 中国科学院计算机网络信息中心 | Security postures intelligent analysis method, device and the storage medium of mobile terminal |
CN109299143B (en) * | 2018-11-28 | 2022-03-22 | 重庆邮电大学 | Knowledge fast indexing method of data interoperation test knowledge base based on Redis cache |
CN109756381B (en) * | 2019-02-11 | 2022-02-25 | 南方科技大学 | Data center fault positioning method and device, electronic equipment and medium |
CN110336785A (en) * | 2019-05-22 | 2019-10-15 | 北京瀚海思创科技有限公司 | The method for visualizing and storage medium of network attack chain figure |
CN110213108A (en) * | 2019-06-11 | 2019-09-06 | 四川久远国基科技有限公司 | A kind of network security situation awareness method for early warning and system |
CN110442550B (en) * | 2019-07-05 | 2022-02-08 | 北京邮电大学 | Log screen-gathering real-time visualization method and device |
CN110554916B (en) * | 2019-07-31 | 2022-07-29 | 苏宁云计算有限公司 | Distributed cluster-based risk index calculation method and device |
CN110460622B (en) * | 2019-09-12 | 2021-11-16 | 贵州电网有限责任公司 | Network anomaly detection method based on situation awareness prediction method |
CN110716973A (en) * | 2019-09-23 | 2020-01-21 | 杭州安恒信息技术股份有限公司 | Big data based security event reporting platform and method |
CN110764969A (en) * | 2019-10-25 | 2020-02-07 | 新华三信息安全技术有限公司 | Network attack tracing method and device |
CN110855506A (en) * | 2019-11-27 | 2020-02-28 | 国家电网有限公司信息通信分公司 | Safety situation monitoring method and system |
CN111193728B (en) * | 2019-12-23 | 2022-04-01 | 成都烽创科技有限公司 | Network security evaluation method, device, equipment and storage medium |
CN111131253A (en) * | 2019-12-24 | 2020-05-08 | 北京优特捷信息技术有限公司 | Scene-based security event global response method, device, equipment and storage medium |
CN111404879A (en) * | 2020-02-26 | 2020-07-10 | 亚信科技(成都)有限公司 | Visualization method and device for network threats |
CN111562930A (en) * | 2020-04-30 | 2020-08-21 | 深圳壹账通智能科技有限公司 | Upgrading method and system for web application security |
CN111935069B (en) * | 2020-06-17 | 2022-08-26 | 西安理工大学 | Traffic attack visualization characterization method based on time sequence |
CN111787011B (en) * | 2020-07-01 | 2022-03-29 | 公安部第三研究所 | Intelligent analysis and early warning system, method and storage medium for security threat of information system |
CN112532625B (en) * | 2020-11-27 | 2022-09-13 | 杭州安恒信息安全技术有限公司 | Network situation awareness evaluation data updating method and device and readable storage medium |
CN112527879B (en) * | 2020-12-15 | 2024-04-16 | 中国人寿保险股份有限公司 | Kafka-based real-time data extraction method and related equipment |
CN113438123B (en) * | 2021-05-26 | 2022-08-30 | 曙光网络科技有限公司 | Network flow monitoring method and device, computer equipment and storage medium |
CN113596025A (en) * | 2021-07-28 | 2021-11-02 | 中国南方电网有限责任公司 | Power grid security event management method |
CN115102778A (en) * | 2022-07-11 | 2022-09-23 | 深信服科技股份有限公司 | State determination method, device, equipment and medium |
CN115643115B (en) * | 2022-12-23 | 2023-03-10 | 武汉大学 | Industrial control network security situation prediction method and system based on big data |
CN117290413A (en) * | 2023-08-05 | 2023-12-26 | 智参软件科技(上海)有限公司 | Factory number real fusion platform based on SaaS and integration method |
CN116756225B (en) * | 2023-08-14 | 2023-11-07 | 南京展研信息技术有限公司 | Situation data information processing method based on computer network security |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101547445A (en) * | 2008-03-25 | 2009-09-30 | 上海摩波彼克半导体有限公司 | System and method for detecting abnormal incursion based on mobility in mobile communication network |
CN101883017A (en) * | 2009-05-04 | 2010-11-10 | 北京启明星辰信息技术股份有限公司 | System and method for evaluating network safe state |
CN103345514A (en) * | 2013-07-09 | 2013-10-09 | 焦点科技股份有限公司 | Streamed data processing method in big data environment |
US20130283233A1 (en) * | 2012-04-24 | 2013-10-24 | Maria Guadalupe Castellanos | Multi-engine executable data-flow editor and translator |
CN103593609A (en) * | 2012-08-16 | 2014-02-19 | 阿里巴巴集团控股有限公司 | Trustworthy behavior recognition method and device |
CN104767757A (en) * | 2015-04-17 | 2015-07-08 | 国家电网公司 | Multiple-dimension security monitoring method and system based on WEB services |
-
2016
- 2016-01-15 CN CN201610028522.1A patent/CN105681303B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101547445A (en) * | 2008-03-25 | 2009-09-30 | 上海摩波彼克半导体有限公司 | System and method for detecting abnormal incursion based on mobility in mobile communication network |
CN101883017A (en) * | 2009-05-04 | 2010-11-10 | 北京启明星辰信息技术股份有限公司 | System and method for evaluating network safe state |
US20130283233A1 (en) * | 2012-04-24 | 2013-10-24 | Maria Guadalupe Castellanos | Multi-engine executable data-flow editor and translator |
CN103593609A (en) * | 2012-08-16 | 2014-02-19 | 阿里巴巴集团控股有限公司 | Trustworthy behavior recognition method and device |
CN103345514A (en) * | 2013-07-09 | 2013-10-09 | 焦点科技股份有限公司 | Streamed data processing method in big data environment |
CN104767757A (en) * | 2015-04-17 | 2015-07-08 | 国家电网公司 | Multiple-dimension security monitoring method and system based on WEB services |
Non-Patent Citations (3)
Title |
---|
"中国科技网网络安全平台及应用";宋丹劼,等;《科研信息化技术与应用》;20150630;全文 * |
"基于Storm和Hadoop的大数据处理架构的研究";靳永超,等;《现代计算机(专业版)》;20150210;第0-2部分,图1 * |
"网络安全分析中的大数据技术应用";王帅,等;《电信科学,2015年第07期》;20150731;第3-4部分,图1 * |
Also Published As
Publication number | Publication date |
---|---|
CN105681303A (en) | 2016-06-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN105681303B (en) | A kind of network safety situation monitoring of big data driving and method for visualizing | |
CN107196910B (en) | Threat early warning monitoring system, method and deployment framework based on big data analysis | |
Aljawarneh et al. | Anomaly-based intrusion detection system through feature selection analysis and building hybrid efficient model | |
CN106170772B (en) | Network safety system | |
CN108270785A (en) | Knowledge graph-based distributed security event correlation analysis method | |
Fischer et al. | Real-time visual analytics for event data streams | |
Yadranjiaghdam et al. | Developing a real-time data analytics framework for twitter streaming data | |
CN106487596A (en) | Distributed Services follow the tracks of implementation method | |
CN102902813B (en) | Result collection system | |
CN104767757A (en) | Multiple-dimension security monitoring method and system based on WEB services | |
CN102111420A (en) | Intelligent NIPS framework based on dynamic cloud/fire wall linkage | |
CN103927398A (en) | Microblog hype group discovering method based on maximum frequent item set mining | |
CN109902297A (en) | A kind of threat information generation method and device | |
CN108123939A (en) | Malicious act real-time detection method and device | |
CN112765366A (en) | APT (android Package) organization portrait construction method based on knowledge map | |
CN104516954A (en) | Visualized evidence obtaining and analyzing system | |
Shi et al. | Visual analytics of anomalous user behaviors: A survey | |
CN103886508A (en) | Mass farmland data monitoring method and system | |
CN109710767A (en) | Multilingual big data service platform | |
El Arass et al. | Smart SIEM: From big data logs and events to smart data alerts | |
CN114430331A (en) | Network security situation sensing method and system based on knowledge graph | |
Perrochon et al. | Enlisting event patterns for cyber battlefield awareness | |
Campos et al. | Creation and deployment of data mining-based intrusion detection systems in oracle database l0g | |
CN113938401A (en) | Naval vessel network security visualization system | |
US20230065398A1 (en) | Cygraph graph data ingest and enrichment pipeline |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |