CN109977125A - A kind of big data safety analysis plateform system based on network security - Google Patents
A kind of big data safety analysis plateform system based on network security Download PDFInfo
- Publication number
- CN109977125A CN109977125A CN201910280131.2A CN201910280131A CN109977125A CN 109977125 A CN109977125 A CN 109977125A CN 201910280131 A CN201910280131 A CN 201910280131A CN 109977125 A CN109977125 A CN 109977125A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- analysis
- network security
- mining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
A kind of big data safety analysis plateform system based on network security, including data acquisition module, data memory module, data mining analysis module and data visualize module；The data acquisition module is connect with the data memory module；The data memory module is connect with the data mining analysis module；The data mining analysis module is connect with the data visualization display module；The data acquisition module is configured as acquisition structural data, unstructured data and semi-structured data；The data memory module is configured as storing the data of the data collecting module collected；The data mining analysis module is configured as and mining analysis for statistical analysis to data；The data visualization display module is configured as to user feedback data.Present invention reduces data carrying costs, improve data storage capacity, improve Network Safety Analysis treatment effeciency.
Description
Technical field
The present invention relates to technical field of network security more particularly to a kind of big data safety analysis based on network security are flat
Platform system.
Background technique
Network security refers to that the data in the hardware, software and its system of network system are protected, not because accidental or
The reason of person's malice and by destruction, change or leakage, system can be continuous, reliable and be normally run, in network service not
It is disconnected.Network analysis system, which is one, allows network manager, can be in various network security problems, the network management suited the remedy to the case
Scheme, it detects the data of transmission all in network, is analyzed, is diagnosed, and helps user to exclude network contingency, evades safety
Risk improves network performance, increases network availability value.
Big data (big data), refer to can not be captured within the scope of certain time with conventional software tool, manage and
The data acquisition system of processing is to need new tupe that could have stronger decision edge, see clearly discovery power and process optimization ability
Magnanimity, high growth rate and diversified information assets.Big data has the characteristics that 5V, i.e. Volume (a large amount of), Velocity are (high
Speed), Variety (multiplicity), Value (low value density) and Veracity (authenticity).Big data processing mainly includes following
Several stages: acquisition, importing/pretreatment, statistics/analysis and excavation.
Rapid development of information technology, the network architecture is increasingly sophisticated, and network security data amount of analysis speedup and increment are huge, number
According to abundance, data category multiplicity, for promptly and accurately detect security risk and attack, information need to be improved and adopted
The rate of collection and transmission；Have the shortcomings that carrying cost height and amount of storage are small using traditional structured database storing data,
And the inefficiency of analysis and complex query is carried out to mass data, it is difficult to meet Network Safety Analysis demand at this stage.
Summary of the invention
(1) goal of the invention
To solve technical problem present in background technique, the present invention proposes a kind of big data safety based on network security
Analysis platform system reduces data carrying cost, improves data storage capacity, improves Network Safety Analysis treatment effeciency.
(2) technical solution
To solve the above problems, the present invention provides a kind of big data safety analysis plateform system based on network security,
Module is visualized including data acquisition module, data memory module, data mining analysis module and data；
The data acquisition module is connect with the data memory module；The data memory module and the data mining
Analysis module connection；The data mining analysis module is connect with the data visualization display module；
The data acquisition module is configured as acquisition structural data, unstructured data and semi-structured data；
The data memory module is configured as storing the data of the data collecting module collected；
The data mining analysis module is configured as and mining analysis for statistical analysis to data；
The data visualization display module is configured as to user feedback data.
Preferably, the offline statistical analysis of data is realized using Hive；The real-time online point of data is realized using Storm
Analysis.
Preferably, it for acquisition, polymerization and the transmission of distributed and High Availabitity massive logs, is realized using Flume；It is right
In the processing for enlivening stream data, cached using Kafka；Storage analysis for real time data, is realized using Storm and is divided
Cloth real-time message processing.
Preferably, it is stored using the HDFS distributed file system with high fault tolerance and high-throughput data access characteristics
The mass data of the data collecting module collected；The data storage management of efficient big data quantity is carried out using Hbase.
Preferably, using MapReduce carries out data division, calculating task scheduling and distributed computing；Using Hive into
The statistical analysis of row data；It is carried out using Mahout based on HadoopMachine learning and data mining.
Above-mentioned technical proposal of the invention has a following beneficial technical effect: data collecting module collected daily record data and
Flow information data, and the data of acquisition are transmitted to data memory module；Data memory module stores different type and purposes
Daily record data and flow information data, including for query and search log data and flow information data, for counting
According to the daily record data and flow information data after the standardization of mining analysis and the log number analyzed in real time for data
According to flow information data, the data of storage are transmitted to data mining analysis module by data memory module；Data mining analysis
Module carries out parallel batching, statistical analysis and mining analysis, data mining analysis module to data and passes data analysis result
Transport to data visualization display module；Data analysis result is fed back to user by data visualization display module.To sum up, of the invention
Data carrying cost is reduced, data storage capacity is improved, improves Network Safety Analysis treatment effeciency.
Detailed description of the invention
Fig. 1 is the structural schematic diagram of the big data safety analysis plateform system proposed by the present invention based on network security.
Specific embodiment
In order to make the objectives, technical solutions and advantages of the present invention clearer, With reference to embodiment and join
According to attached drawing, the present invention is described in more detail.It should be understood that these descriptions are merely illustrative, and it is not intended to limit this hair
Bright range.In addition, in the following description, descriptions of well-known structures and technologies are omitted, to avoid this is unnecessarily obscured
The concept of invention.
As shown in Figure 1, a kind of big data safety analysis plateform system based on network security proposed by the present invention, including number
Module is visualized according to acquisition module, data memory module, data mining analysis module and data；
The data acquisition module is connect with the data memory module；The data memory module and the data mining
Analysis module connection；The data mining analysis module is connect with the data visualization display module；
The data acquisition module is configured as acquisition structural data, unstructured data and semi-structured data；
The data memory module is configured as storing the data of the data collecting module collected；
The data mining analysis module is configured as and mining analysis for statistical analysis to data；
The data visualization display module is configured as to user feedback data.
In the present invention, data collecting module collected daily record data and flow information data, and the data of acquisition are transmitted to
Data memory module；Data memory module stores the daily record data and flow information data of different type and purposes, including is used for
The log data and flow information data of query and search, for the daily record data after the standardization of data mining analysis
The daily record data and flow information data analyzed in real time with flow information data and for data, data memory module will store
Data be transmitted to data mining analysis module；Data mining analysis module to data carry out parallel batching, statistical analysis and
Data analysis result is transmitted to data visualization display module by mining analysis, data mining analysis module；Data visualization exhibition
Show that data analysis result is fed back to user by module.To sum up, present invention reduces data carrying costs, improve data storage
Amount, improves Network Safety Analysis treatment effeciency.
In an alternative embodiment, the offline statistical analysis of data is realized using Hive；Data are realized using Storm
On-line analysis.
It should be noted that platform is broadly divided into two classes to the analysis of daily record data and flow information data: offline statistics
Analysis and on-line analysis, offline statistical analysis realize that on-line analysis is realized by Storm by Hive.
In an alternative embodiment, it for acquisition, polymerization and the transmission of distributed and High Availabitity massive logs, adopts
It is realized with Flume；For enlivening the processing of stream data, cached using Kafka；Storage analysis for real time data,
Distributed real-time message processing is realized using Storm.
It should be noted that Flume supports to customize Various types of data sender in log system, for collecting data；Together
When, Flume, which is provided, carries out simple process to data, and writes the ability of various data receivings；Flume can will be applied and be generated
Data store such as the HDFS and HBase into any pooled storage；When the speed for collecting data is more than that data will be written
When, the information of collection is very big, be even more than the write-in data capability of system, at this point, Flume can in data producer and
It is adjusted between data receptacle, guarantees that it can provide stable data therebetween；Flume can provide context routing
Feature；The pipeline of Flume is based on affairs, ensure that consistency of the data in transmission and reception；Flume is reliable, fault-tolerant
Property is high, scalable, manageable and customized.
Kafka is that a kind of distributed post of high-throughput subscribes to message system, its purpose be by Hadoop and
Row load mechanism provides real-time message by cluster come the Message Processing unified on offline and line.
Storm is free and open source a distributed real time computation system, can accomplish reliably to handle using Storm
Unlimited data flow, as Hadoop batch processing big data, Storm can be can be used with real-time processing data, Storm to be appointed
What programming language, Storm have a characteristic that simple programming, high-performance, distribution, expansible, fault-tolerant and message save.
In an alternative embodiment, it is distributed using with the HDFS of high fault tolerance and high-throughput data access characteristics
Formula file system stores the mass data of the data collecting module collected；The data of efficient big data quantity are carried out using Hbase
Storage management.
It should be noted that HDFS has the characteristics that high fault tolerance, it is designed to be deployed on cheap hardware, it passes through height
The characteristic of handling capacity carrys out the data of access application, is suitble to the application program of super large data set；HDFS passes through with data text
The NameSpace of the back end management file system of part store function.
HBase is a high reliability, high-performance, towards column, telescopic distributed memory system, HBase relationship number
According to library tool there are two types of feature, first feature is that it is the database for being suitable for unstructured data storage, second
Feature be it is per-column rather than based on capable mode；All data files in HBase are stored in Hadoop HDFS
It mainly include two types: Hfile and HLog File in file system.
In an alternative embodiment, data division, calculating task scheduling and distributed meter are carried out using MapReduce
It calculates；The statistical analysis of data is carried out using Hive；Machine learning and data mining based on Hadoop are carried out using Mahout.
It should be noted that MapReduce is computation model, frame and platform towards big data parallel processing, it has
Three layers of meaning: first layer meaning is that MapReduce is the high performance parallel computation platform based on cluster, it is realizing one
It, can be only with common commercial server realization when a distribution and parallel computing trunking comprising tens of, hundreds of to many thousands of nodes；
Second layer meaning is that MapReduce is a parallel computation and runs software frame, it is by providing a huge but design
Superior parallel computation software frame is appointed to be automatically performed parallelization processing, automatic division calculating data and the calculating of calculating task
Business distributes and executes automatically task on clustered node and collect calculated result, by distributed data storage, data communication, appearance
The ins and outs for many system bottoms that the parallel computations such as fault reason are related to transfers to system to be responsible for processing；Third layer meaning is
MapReduce is a Parallel programming model and method, with two function programming realities of Map (mapping) and Reduce (reduction)
Now basic parallel computation task provides abstract operation and multiple programming interface, simply and easily to complete extensive number
According to programming and calculation processing.
Hive is built upon the data warehouse base frame on Hadoop, have a characteristic that support creation index with
Semantic inspection is executed when optimizing data query, storage files in different types, storing metadata in relational database to reduce inquiry
Time for looking into, can directly using the data being stored in Hadoop system, support user to extend looking into for UDF function and class SQL
Inquiry mode.
Mahout includes a variety of realizations, including cluster, classification, recommendation filtering and frequent subitem excavate, by using
The library Apache Hadoop, Mahout can be effectively extended in cloud, and in terms of creating intelligent application, Mahout can be effective
Ground help developer realizes the creation of program more conveniently.
It should be understood that above-mentioned specific embodiment of the invention is used only for exemplary illustration or explains of the invention
Principle, but not to limit the present invention.Therefore, that is done without departing from the spirit and scope of the present invention is any
Modification, equivalent replacement, improvement etc., should all be included in the protection scope of the present invention.In addition, appended claims purport of the present invention
Covering the whole variations fallen into attached claim scope and boundary or this range and the equivalent form on boundary and is repairing
Change example.
Claims (5)
1. a kind of big data safety analysis plateform system based on network security, which is characterized in that including data acquisition module, number
Module is visualized according to memory module, data mining analysis module and data；
The data acquisition module is connect with the data memory module；The data memory module and the data mining analysis
Module connection；The data mining analysis module is connect with the data visualization display module；
The data acquisition module is configured as acquisition structural data, unstructured data and semi-structured data；
The data memory module is configured as storing the data of the data collecting module collected；
The data mining analysis module is configured as and mining analysis for statistical analysis to data；
The data visualization display module is configured as to user feedback data.
2. the big data safety analysis plateform system according to claim 1 based on network security, which is characterized in that use
The offline statistical analysis of Hive realization data；The on-line analysis of data is realized using Storm.
3. the big data safety analysis plateform system according to claim 2 based on network security, which is characterized in that for
Acquisition, polymerization and the transmission of distributed and High Availabitity massive logs, are realized using Flume；For enlivening the place of stream data
Reason, is cached using Kafka；Storage analysis for real time data realizes distributed real-time message processing using Storm.
4. the big data safety analysis plateform system according to claim 1 based on network security, which is characterized in that use
The data acquisition module is stored with the HDFS distributed file system of high fault tolerance and high-throughput data access characteristics to adopt
The mass data of collection；The data storage management of efficient big data quantity is carried out using Hbase.
5. the big data safety analysis plateform system according to claim 1 based on network security, which is characterized in that use
MapReduce carries out data division, calculating task scheduling and distributed computing；The statistical analysis of data is carried out using Hive；It adopts
Machine learning and data mining based on Hadoop are carried out with Mahout.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910280131.2A CN109977125A (en) | 2019-04-09 | 2019-04-09 | A kind of big data safety analysis plateform system based on network security |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910280131.2A CN109977125A (en) | 2019-04-09 | 2019-04-09 | A kind of big data safety analysis plateform system based on network security |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109977125A true CN109977125A (en) | 2019-07-05 |
Family
ID=67083580
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910280131.2A Pending CN109977125A (en) | 2019-04-09 | 2019-04-09 | A kind of big data safety analysis plateform system based on network security |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109977125A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111090646A (en) * | 2019-10-21 | 2020-05-01 | 中国科学院信息工程研究所 | Electromagnetic data processing platform |
| CN111753008A (en) * | 2020-06-30 | 2020-10-09 | 珠海迈越信息技术有限公司 | Set top box viewing method and system based on big data analysis |
| CN112163060A (en) * | 2020-09-16 | 2021-01-01 | 安徽龙运智能科技有限公司 | System for processing mass GPS data by big data technology |
| CN113742413A (en) * | 2021-09-10 | 2021-12-03 | 湖南强智科技发展有限公司 | High-accuracy big data analysis system based on multi-form processing |
| CN114268503A (en) * | 2021-12-24 | 2022-04-01 | 湘潭大学 | Network space data security situation sensing and early warning method and system |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105553957A (en) * | 2015-12-09 | 2016-05-04 | 国家电网公司 | Network safety situation awareness early-warning method and system based big data |
| CN107145587A (en) * | 2017-05-11 | 2017-09-08 | 成都四方伟业软件股份有限公司 | A kind of anti-fake system of medical insurance excavated based on big data |
| CN107181612A (en) * | 2017-05-08 | 2017-09-19 | 深圳市众泰兄弟科技发展有限公司 | A kind of visual network method for safety monitoring based on big data |
| CN107196910A (en) * | 2017-04-18 | 2017-09-22 | 国网山东省电力公司电力科学研究院 | Threat early warning monitoring system, method and the deployment framework analyzed based on big data |
| CN107515927A (en) * | 2017-08-24 | 2017-12-26 | 深圳市云房网络科技有限公司 | A kind of real estate user behavioural analysis platform |
| CN109150870A (en) * | 2018-08-14 | 2019-01-04 | 中国电子科技集团公司电子科学研究院 | Network safety situation analysis and application method and system |
-
2019
- 2019-04-09 CN CN201910280131.2A patent/CN109977125A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104820670A (en) * | 2015-03-13 | 2015-08-05 | 国家电网公司 | Method for acquiring and storing big data of power information |
| CN105553957A (en) * | 2015-12-09 | 2016-05-04 | 国家电网公司 | Network safety situation awareness early-warning method and system based big data |
| CN107196910A (en) * | 2017-04-18 | 2017-09-22 | 国网山东省电力公司电力科学研究院 | Threat early warning monitoring system, method and the deployment framework analyzed based on big data |
| CN107181612A (en) * | 2017-05-08 | 2017-09-19 | 深圳市众泰兄弟科技发展有限公司 | A kind of visual network method for safety monitoring based on big data |
| CN107145587A (en) * | 2017-05-11 | 2017-09-08 | 成都四方伟业软件股份有限公司 | A kind of anti-fake system of medical insurance excavated based on big data |
| CN107515927A (en) * | 2017-08-24 | 2017-12-26 | 深圳市云房网络科技有限公司 | A kind of real estate user behavioural analysis platform |
| CN109150870A (en) * | 2018-08-14 | 2019-01-04 | 中国电子科技集团公司电子科学研究院 | Network safety situation analysis and application method and system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111090646A (en) * | 2019-10-21 | 2020-05-01 | 中国科学院信息工程研究所 | Electromagnetic data processing platform |
| CN111090646B (en) * | 2019-10-21 | 2023-07-28 | 中国科学院信息工程研究所 | Electromagnetic data processing platform |
| CN111753008A (en) * | 2020-06-30 | 2020-10-09 | 珠海迈越信息技术有限公司 | Set top box viewing method and system based on big data analysis |
| CN112163060A (en) * | 2020-09-16 | 2021-01-01 | 安徽龙运智能科技有限公司 | System for processing mass GPS data by big data technology |
| CN113742413A (en) * | 2021-09-10 | 2021-12-03 | 湖南强智科技发展有限公司 | High-accuracy big data analysis system based on multi-form processing |
| CN114268503A (en) * | 2021-12-24 | 2022-04-01 | 湘潭大学 | Network space data security situation sensing and early warning method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109977125A (en) | A kind of big data safety analysis plateform system based on network security | |
| CN104820670B (en) | A kind of acquisition of power information big data and storage method | |
| Gupta et al. | Cloud computing and big data analytics: what is new from databases perspective? | |
| CN111435344B (en) | Big data-based drilling acceleration influence factor analysis model | |
| JP2022078130A (en) | System and method for mata data-driven external interface generation of application programming interface | |
| CN109272155A (en) | A kind of corporate behavior analysis system based on big data | |
| CN103631882B (en) | Semantization service generation system and method based on graph mining technique | |
| CN106339509A (en) | Power grid operation data sharing system based on large data technology | |
| CN107943668A (en) | Computer server cluster daily record monitoring method and monitor supervision platform | |
| CN106815338A (en) | A kind of real-time storage of big data, treatment and inquiry system | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN110347719A (en) | A kind of enterprise's foreign trade method for prewarning risk and system based on big data | |
| CN105139281A (en) | Method and system for processing big data of electric power marketing | |
| CN112116488A (en) | Water conservancy big data comprehensive maintenance system | |
| CN106571960A (en) | Log collection and management system and log collection and management method | |
| Fu et al. | Real-time data infrastructure at uber | |
| CN109754219A (en) | A kind of big data management and analysis platform system based on city management | |
| CN102902813A (en) | Log collection system | |
| CN112148718A (en) | Big data support management system for city-level data middling station | |
| Caldarola et al. | Big data: A survey-the new paradigms, methodologies and tools | |
| Roth et al. | Event data warehousing for complex event processing | |
| CN107103064A (en) | Data statistical approach and device | |
| CN112181960A (en) | Intelligent operation and maintenance framework system based on AIOps | |
| CN111459900B (en) | Big data life cycle setting method, device, storage medium and server | |
| Mesiti et al. | Towards a user-friendly loading system for the analysis of big data in the internet of things |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190705 |