CN105207826A - Security attack alarm positioning system based on Spark big data platform of Tachyou - Google Patents
Security attack alarm positioning system based on Spark big data platform of Tachyou Download PDFInfo
- Publication number
- CN105207826A CN105207826A CN201510695048.3A CN201510695048A CN105207826A CN 105207826 A CN105207826 A CN 105207826A CN 201510695048 A CN201510695048 A CN 201510695048A CN 105207826 A CN105207826 A CN 105207826A
- Authority
- CN
- China
- Prior art keywords
- alarm
- security attack
- information
- submodule
- security
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Alarm Systems (AREA)
Abstract
The invention discloses a security attack alarm positioning system based on a Spark big data platform of Tachyou and belongs to the field of big data and information security. The system comprises an acquisition module, a security attack alarm positioning module and a view module. The acquisition module gathers log information in an enterprise information system, preprocesses and pushes the log information to the security attack alarm positioning module in real time, the security attack alarm positioning module performs real-time analysis to generate alarm information, and sends the alarm information to a view front end page. Compared with the prior art, the security attack alarm positioning system can solve the problem that a large number of logs are difficult to analyze in real time and has excellent practicality and value of popularization and application.
Description
Technical field
The present invention relates to the technical field of information security, the large data platform of Spark, Flume log collection, kafka data interchange platform, HDFS and Tachyou distributed memory file system, refer more particularly to security attack alarm navigation system.
Background technology
The English abbreviation comprised in the present invention is as follows:
SOC:SecurityOperationCenter security management center
IDS:IntrusionDetectionSystems intruding detection system
DDOS:DDoS:DistributedDenialofService distributed denial of service attack
MIS:ManagementInformationSystem management information system
DMZ:demilitarizedzone isolated area or demilitarized zone
JMS:JavaMessageServiceJava messenger service
APP:Application application program
SNMP:SimpleNetworkManagementProtocol Simple Network Management Protocol
HDFS:HadoopDistributeFileSystemHadoop distributed file system
ODBC:OpenDatabaseConnectivity Open Database Connection
WMI:WindowsManagementInstrumentationWindows management regulation
Safety in production is always the prerequisite ensureing that work in every is carried out in order, is also the rejection index of examination leading cadres at various levels.Network and information security operation and maintenance system is the important component part of all kinds of enterprise safety operation work.Logistics networks runs efficiently and stably, is the basis of all market management activities of enterprise and normal operation.
Along with the construction of all kinds of enterprise information system and perfect, effectively raise labor productivity, reduce operation cost.Once there is security incident or break down or forming property bottleneck in each operation system of enterprise, can not Timeliness coverage, in time process, recover in time, certainly will directly cause carrying the operation of all business thereon, affect the normal operation order of enterprise, business event can not normally be carried out.Therefore, the safety guarantee implemented for Government and enterprise IT basis just seems especially important.
Along with the Government and enterprise level of informatization improves constantly.Contact more and more closer between each operation system, exchanges data is more and more frequent, each system has complex network or logic to connect, there is mass data to exchange, even a fault can cause and become enterprise's the whole network fault, any or a kind of operation system start a leak and infect virus or under attack, will involve rapidly other operation system and network, even cause enterprise's the whole network paralysis.
Although the information security technology system of some enterprises begins to take shape at present, but information safety operation and maintenance management system needs further sound to improve and perfect, managerial ability also has to be strengthened, the degree of depth lacking potential safety hazard is excavated and based on the safety analysis of large data platform, security attack alarm location and analysis tool few.Owing to lacking macroscopical thinking of security system building, there is no-man's-land in safety management, responsibility does not have effective execution.
At present, there is following problem in all kinds of enterprise information security operation management platform:
1, various safety information product and the network equipment wide in variety, distribution is wide, lacks unified data analysis management;
2, the knowledge base disunity of safety information product and the network equipment, lacks unified solution;
3, security responsibility is unclear, and specific responsibility is not implemented completely;
4, information safety operation and maintenance management evaluation is not careful, lacks the index that part is necessary and crucial;
5, the analysis that between different safety means event, the event of even same safety means lacks high-grade intelligent more associates with convergence, causes warning information huge, is not easy to the analysis of potential safety hazard and pinpoints the problems, preventing trouble before it happens;
6, information security events reports not in time, and not in time, treatment effeciency is low for failure diagnosis, weak effect;
7, the leak of information security events and assets does not carry out necessary association analysis, causes a lot of event not have further treatment and analysis;
8, cannot carry out auditing and checking easily for the safety problem of terminal;
9, occur that emergency does not have good early warning and handling process;
10, security attack alarm location and analysis tool few;
There is the business and network that enterprise built up in the problems referred to above, becomes the obstacle that lifting is stablized in enterprise's service security operation management from now on to some extent.
For this reason, information-based means how are utilized to improve enterprise security operation management benefit, solve the safe operation management hidden danger existing for each system of enterprise, and design a security attack alarm navigation system based on large data platform, optimize enterprise information security and administer and maintain work, make it can provide specialty with high efficiency information safety operation and maintenance management service for all kinds of enterprise, namely become the important topic that especially information safety operation and maintenance management design must solve.
Summary of the invention
The present invention, after the defect analyzing above-mentioned all kinds of enterprise information security operation management and deficiency, proposes the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou.
Technical assignment of the present invention realizes in such a way: the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou, comprises acquisition module, security attack alarm locating module, views module.
Described acquisition module, for gathering the daily record of various equipment, and carries out preliminary treatment and real-time Transmission to locating module (SparkStreaming), and preliminary treatment comprises daily record and filters, merges and standardized format, unified daily record specification.
Described security attack alarm locating module, for the daily record collected, carries out real-time analysis and obtains warning information.
Described views module, by the information inquiry in MySQL database with represent, provides the inquiry and analysis of warning information and log information.
Said system passes through acquisition module, gather log information in enterprise information system and real time propelling movement to security attack alarm locating module, produce warning information by the real-time analysis of security attack alarm locating module and be sent to the front end page of views module, and providing attack to trace to the source, put to the proof and inquire about.
Preferably, acquisition module is developed and the distributed information system of increasing income by LinkedIn by flume being integrated into kafka(kafka) in realize the real-time Transmission of log collection, log integrity and daily record.It can gather syslog daily record, monitoring file daily record and the daily record of TCP/UDP port etc., and, can dock with SparkStreaming well, realize pretreated log information real-time Transmission to locating module.
Security attack alarm locating module obtains warning information to daily record real-time analysis, and is transferred to views module after receiving log information; Meanwhile, the log information after standardization is stored in HDFS, and warning information is stored in MySQL database.
Described security attack alarm locating module, comprises off-line association submodule, online association submodule, alarm generates submodule and attack type finds submodule.
Described off-line association submodule, utilizes the history log information be stored in HDFS to build alarm correlation analysis model, and online updating knowledge base.
Described online association submodule, utilizes knowledge base to carry out online association analysis.
Described attack type finds submodule, warning information is carried out cluster analysis, finds its characteristic sum challenge model.
Described knowledge base, at least comprises:
1, according to the partial content of wall scroll daily record as warning information; Such as, the login occurred in Windows daily record, startup and shutdown can, as warning information, can use ElasticSearch to carry out keyword search;
2, according to the frequency that special event in the unit interval occurs; Using this special event as warning information; Such as, occur in 1 minute in Windows daily record that the situation of 3 user cipher mistakes can as a Brute Force;
3, the association analysis between many device logs take analysis result as warning information; Such as, many target ip address are identical, and the daily record that source IP address is different, then can as a DDOS attack;
Described security attack alarm locating module, stores warning information in MySQL database, and by the daily record relevant to warning information also in this MySQL database.By the analysis to the log information be stored in MySQL database, the information such as attack source, attack path can be obtained further.
Compared with prior art, the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou of the present invention, has following outstanding beneficial effect:
1, the distributed structure/architecture of large data platform, is easy to expansion and reduction, can tackle enterprise network scale change and change its own system size to reach effective utilization of resource, also solve the drawback that prior art is difficult to process massive logs;
2, positioning function improves the accuracy of alarm and eliminates wrong report, and provides detailed warning information analysis, facilitates its work;
3, utilize large data technique to carry out data mining and machine learning, the magnanimity history log information collected can be effectively utilized, by the off-line association analysis be combined with existing knowledge base, can automatic expansion knowledge base;
4, in large data system, usually can encounter a problem, whole large data are made up of subsystems, and data need high-performance, not arrheaing of low delay in subsystems to turn.Traditional enterprise information system is not be applicable to very much large-scale data processing.In order to settle application on site (message) and off-line application (data file, daily record) simultaneously, Kafka has just occurred; Further, can FlumeNG be integrated in Kafka, utilize many source and sink assemblies that FlumeNG is built-in, realize the log collection of various equipment, and be transferred to the large data platform of Spark or internal memory distributed file system Tachyon or HDFS or MySQL database or view front end by Kafka and carry out showing and inquiring about, etc.;
5, store intermediate object program by Tachyon, avoid data and fall on disk, share to realize internal storage data.Meanwhile, walk around HDFS and can reduce the therefore disk caused and network I/O.Moreover to be data cachedly all stored in Tachyon because all, the JVM collapse caused by Spark task abnormity can not cause loss of data.
Accompanying drawing explanation
Fig. 1 is the Organization Chart of the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou of the present invention;
Fig. 2 is the flow chart of the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou of the present invention;
Embodiment
Here be with reference to the accompanying drawings with example to further description of the present invention:
The security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou of the present invention, by flume being incorporated into the daily record of various equipment in kafka Distributed data exchange systematic collection corporate environment, and preliminary treatment in real time and be transferred to locating module.Locating module provides knowledge base to analyze in real time, and analysis result is pushed to foreground, and provides warning information to trace to the source and put to the proof function.Its framework is as shown in Figure 1: (1) acquisition module, is made up of kafka Distributed data exchange system.(2) security attack alarm locating module, is made up of SparkStreaming; (3) views module, provides the inquiry of warning information and analysis, log information and analysis.
Described acquisition module, is the prerequisite of security attack alarm location, is the bottom of whole system, can utilizes and flume is incorporated into kafka to realize acquisition module.The daily record of the various operating systems in this module primary responsibility collection enterprise network, router, switch, safety means, and preliminary treatment in real time and be transferred to locating module, namely in SparkStreaming system, it is the system of a High Availabitity, highly reliable, distributed massive logs collection, polymerization and transmission.ApacheKafka is a distributed post-subscribe message system.It is developed by LinkedIn company at first, becomes a part for Apache project afterwards.Log collection, realizes by being incorporated in Kafka by ApacheFlume.Use Kafka that processing procedure is postponed lower, more easily support multi-data source and distributed data processing.There is provided same performance efficiently compared with the system centered by daily record such as Flume, Kafka and ensure because copy the higher durability caused, and lower end-to-end delay.The daily record one by one flume collected by Kafka or the message flow of event, after preliminary treatment, and after providing in real time, the SparkStreaming streaming Computational frame of docking processes, and be transferred to view front end in real time by kafka and show.
Further, can Flume be integrated in Kafka, utilize many source and sink assemblies that Flume is built-in, realize the log collection of various equipment, and be transferred to the large data platform of Spark or internal memory distributed file system Tachyon or HDFS or MySQL database or view front end by Kafka and carry out showing and inquiring about, etc.
Flume is integrated in Kafka, can realize by configuring as follows: as the Flume end configuration of Producer, be such as wherein source data source with syslog, sink is kafka, and configure as the Flume end of consumer, source is Kafka, and sink is logger; Further, spark-streaming-kafka_2.10, spark-streaming-flume_2.10 be added to it to rely on.
Kafka is integrated in Spark, according to the announced version of current Spark (such as, Spark1.3), supports two kinds of methods: a kind of is support the method based on receiver, and another kind is direct method (not having receiver).Kafka is as Distributed Message Queue, existing very outstanding throughput, there are again higher reliability and autgmentability, adopt Kafka to transmit middleware as daily record and receive daily record, capture the daily record that in enterprise information system, various equipment sends, meanwhile, accept the request of SparkStreaming, daily record is sent to SparkStreaming cluster according to the order of sequence; Docked with Kafka cluster by SparkStreaming cluster, SparkStreaming obtains daily record one by one and to go forward side by side row relax from Kafka cluster.SparkStreaming can obtain data in real time and is stored in inner available memory space from Kafka cluster.Show and page request for the ease of front end, process the result obtained and will be written in MySQL database.
Compared to traditional process framework, framework Kafka being incorporated into SparkStreaming has following advantage: the efficient and low delay guaranteed of (1) the Spark framework real-time/quasi real time property of SparkStreaming operation; (2) what utilize Spark framework to provide enriches API and high flexibility, can write out comparatively complicated algorithm with simplifying; (3) height of programming model unanimously makes left-hand seat SparkStreaming quite easy, also can ensure multiplexing in process in real time and batch processing of service logic simultaneously.
Tachyon is a distributed memory file system, compatible with Spark.It is while alleviating Spark memory pressure, also gives the ability of Spark internal memory rapid, high volume reading and writing data.Tachyon separates the function of memory from Spark, makes Spark can of more absorbed calculating itself, in the hope of reaching higher execution efficiency and real-time performance by the thinner division of labor, and supports the multiple storage modes such as HDFS.
Described security attack alarm locating module, receive the log information that acquisition module sends over, on the one hand, the daily record after standardization is stored in HDFS, on the other hand, according to knowledge base, real-time analysis carried out to daily record and obtain warning information and be transferred to front end at once.MySQL database stores warning information and daily record this week.Knowledge base is such as: (1) according to the partial content of wall scroll daily record as warning information; Such as, the login occurred in Windows daily record, startup and shutdown can as warning information.(2) according to the frequency that special event in the unit interval occurs; Using this special event as warning information; Such as, occur in 3 minutes in Windows daily record that the situation of 5 user cipher mistakes can as a Brute Force.(3) association analysis between many device logs take analysis result as warning information; Such as, many target ip address are identical, and the daily record that source IP address is different, then can as a DDOS attack.The warning information produced is pushed to front end, timely alarm in real time.
Preferably, described off-line association submodule, by the large data platform of Spark based on Tachyou to the support of data mining, machine learning and figure computing technique.Off-line analysis is carried out to the history log be stored in HDFS, and is combined the knowledge entry producing and do not comprise in new knowledge base with knowledge base, find unknown attack.
Preferably, described online association submodule, carries out real-time analysis according to knowledge base to daily record.
Preferably, described alarm generates submodule, obtains warning information, and be transferred to front end at once, timely alarm by association submodule.
Preferably, described attack type finds submodule, by association analysis, finds its feature, finds attack type.
As shown in Figure 2, the flow chart of the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou of the present invention is as follows:
(1) Real-time Collection daily record; Be integrated into by utilizing flume the log information that kafka gathers various kinds of equipment, and by journal format standardization, the equipment gathered comprises various operating system, router, switch, safety means.The daily record gathered, on the one hand, sends to HDFS to store, and on the other hand, sends to the large data platform of Spark based on Tachyou, carries out attack alarm location;
(2) off-line association; By the large data platform of Spark based on Tachyou to the support of data mining, machine learning and figure computing technique.Off-line analysis is carried out to the history log be stored in HDFS, and is combined the knowledge entry producing and do not comprise in new knowledge base with knowledge base, find unknown attack model, expansion knowledge base;
(3) online association; The large data platform of Spark based on Tachyou, according to knowledge base, carries out real-time analysis to the daily record from kafka, generally can produce many warning information;
(4) alarm is compared; These many alarms, according to knowledge base, in order to make produced warning information be conducive to location, can compare according to certain algorithm by the large data platform of Spark, and by comparative result stored in knowledge base, storehouse of refreshing one's knowledge;
(5) alarm level; The result produced according to " alarm is compared " to determine the rank of the order of severity of alarm, and by result stored in knowledge base, storehouse of refreshing one's knowledge; In general, alarm level is higher, then its destructiveness is larger;
(6) Alert aggregation; According to certain algorithm, alarm is carried out polymerization classification, generate multiple cluster;
(7) attack type finds; By multiple clusters that Alert aggregation generates, namely generate than simple multiple alarm graph structure (or illustraton of model) before, a challenge model figure is exactly the expression of an alarm figure, and storehouse of refreshing one's knowledge;
(8) view shows; The warning information that real-time reception locating module sends over, and alarm, and log query function etc. is provided.
The foregoing is only preferred embodiment of the present invention, be not used for limiting practical range of the present invention; Every equivalence done according to the present invention changes and amendment, is all regarded as the scope of the claims of the present invention and contains.
Claims (3)
1. the invention provides the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou, its feature is, comprises acquisition module, security attack alarm locating module and views module:
● described acquisition module, for gathering the daily record of various equipment, and preliminary treatment in real time, standardize daily record and be transferred to locating module;
● described security attack alarm locating module, obtains warning information according to knowledge base real-time analysis;
● described views module, by the inquiry of information in MySQL database, provides the inquiry and analysis of warning information and log information.
2. the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou as claimed in claim 1, its feature is, described acquisition module, is integrated in kafka realizes log collection, preliminary treatment and real-time Transmission by utilizing Flume.
3. the security attack alarm navigation system of the large data platform of a kind of Spark based on Tachyou as claimed in claim 1, its feature is, described security attack alarm locating module, comprises off-line association submodule, online association submodule, alarm generates submodule and attack type finds submodule:
● described off-line association submodule, carries out off-line analysis to the history log be stored in HDFS, and is combined the knowledge entry that produces and do not comprise in new knowledge base with knowledge base to the storehouse that expands knowledge, discovery unknown attack model;
● described online association submodule, carries out real-time analysis according to knowledge base to daily record;
● described alarm generates submodule, obtains warning information, and is transferred to front end at once, timely alarm by online association submodule;
● described attack type finds submodule, by association analysis, finds its feature, finds attack type.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510695048.3A CN105207826A (en) | 2015-10-26 | 2015-10-26 | Security attack alarm positioning system based on Spark big data platform of Tachyou |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510695048.3A CN105207826A (en) | 2015-10-26 | 2015-10-26 | Security attack alarm positioning system based on Spark big data platform of Tachyou |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105207826A true CN105207826A (en) | 2015-12-30 |
Family
ID=54955279
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510695048.3A Pending CN105207826A (en) | 2015-10-26 | 2015-10-26 | Security attack alarm positioning system based on Spark big data platform of Tachyou |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105207826A (en) |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
| CN105915377A (en) * | 2016-04-14 | 2016-08-31 | 北京思特奇信息技术股份有限公司 | Flume and Spark streaming integration method and system |
| CN105933186A (en) * | 2016-06-30 | 2016-09-07 | 北京奇虎科技有限公司 | Security detection method, device and system |
| CN105930373A (en) * | 2016-04-13 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Spark streaming based big data stream processing method and system |
| CN106230819A (en) * | 2016-07-31 | 2016-12-14 | 上海交通大学 | A kind of DDoS detection method based on stream sampling |
| CN106254125A (en) * | 2016-08-18 | 2016-12-21 | 南京联成科技发展有限公司 | The method and system of security incident correlation analysiss based on big data |
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN106992872A (en) * | 2016-01-21 | 2017-07-28 | 中国移动通信集团公司 | A kind of method and system of information processing |
| CN107104951A (en) * | 2017-03-29 | 2017-08-29 | 国家电网公司 | The detection method and device of Attack Source |
| CN107220348A (en) * | 2017-05-27 | 2017-09-29 | 郑州云海信息技术有限公司 | A kind of method of data capture based on Flume and Alluxio |
| CN107453882A (en) * | 2016-05-30 | 2017-12-08 | 北京京东尚科信息技术有限公司 | Warning information paradigmatic system and method in a kind of cluster |
| CN107547229A (en) * | 2016-06-29 | 2018-01-05 | 南京联成科技发展股份有限公司 | A kind of implementation method of the safe operation management platform intelligent control based on big data |
| CN107579944A (en) * | 2016-07-05 | 2018-01-12 | 南京联成科技发展股份有限公司 | Based on artificial intelligence and MapReduce security attack Forecasting Methodologies |
| CN108021809A (en) * | 2017-12-19 | 2018-05-11 | 北京明朝万达科技股份有限公司 | A kind of data processing method and system |
| CN108334646A (en) * | 2018-04-11 | 2018-07-27 | 焦点科技股份有限公司 | A kind of link structure optimization method based on frequent browsing sequence |
| CN108737549A (en) * | 2018-05-25 | 2018-11-02 | 江苏联盟信息工程有限公司 | A kind of log analysis method and device of big data quantity |
| CN108763562A (en) * | 2018-06-04 | 2018-11-06 | 广东京信软件科技有限公司 | A kind of construction method based on big data skill upgrading data exchange efficiency |
| CN108964957A (en) * | 2017-05-24 | 2018-12-07 | 中兴通讯股份有限公司 | A kind of method and big data system of data communication service quality monitoring |
| CN109525422A (en) * | 2018-10-31 | 2019-03-26 | 武汉雨滴科技有限公司 | A kind of daily record data method for managing and monitoring |
| CN109861844A (en) * | 2018-12-07 | 2019-06-07 | 中国人民大学 | A kind of cloud service problem fine granularity intelligence source tracing method based on log |
| CN109951463A (en) * | 2019-03-07 | 2019-06-28 | 成都古河云科技有限公司 | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column |
| CN110690984A (en) * | 2018-07-05 | 2020-01-14 | 上海宝信软件股份有限公司 | Spark-based big data weblog acquisition, analysis and early warning method and system |
| CN110879771A (en) * | 2019-11-05 | 2020-03-13 | 北京航空航天大学 | Log analysis system for user anomaly detection based on keyword sequence mining |
| CN110968470A (en) * | 2018-09-28 | 2020-04-07 | 江苏赛融科技股份有限公司 | Operation and maintenance monitoring and aggregation management system |
| CN111131253A (en) * | 2019-12-24 | 2020-05-08 | 北京优特捷信息技术有限公司 | Scene-based security event global response method, device, equipment and storage medium |
| CN111209258A (en) * | 2019-12-31 | 2020-05-29 | 航天信息股份有限公司 | Tax end system log real-time analysis method, equipment, medium and system |
| CN111782477A (en) * | 2020-06-30 | 2020-10-16 | 平安国际智慧城市科技股份有限公司 | Abnormal log monitoring method and device, computer equipment and storage medium |
| CN113542311A (en) * | 2021-09-17 | 2021-10-22 | 成都数默科技有限公司 | Method for detecting and backtracking defect host in real time |
| CN113704069A (en) * | 2021-07-20 | 2021-11-26 | 北京直真科技股份有限公司 | Alarm system fault positioning method based on flash log collection technology |
| CN114760189A (en) * | 2022-03-30 | 2022-07-15 | 深信服科技股份有限公司 | Information determination method, equipment and computer readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127594A (en) * | 2007-10-10 | 2008-02-20 | 杭州华三通信技术有限公司 | A device and method for secure information joint processing |
| US20110035781A1 (en) * | 2009-04-07 | 2011-02-10 | Pratyush Moghe | Distributed data search, audit and analytics |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN103561018A (en) * | 2013-10-30 | 2014-02-05 | 蓝盾信息安全技术股份有限公司 | Intrusion detection real-time analysis system for big data application platform |
| CN104426697A (en) * | 2013-08-29 | 2015-03-18 | 上海斐讯数据通信技术有限公司 | Network fault management system |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Spark-based log auditing and reversed checking system for big data platforms |
-
2015
- 2015-10-26 CN CN201510695048.3A patent/CN105207826A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101127594A (en) * | 2007-10-10 | 2008-02-20 | 杭州华三通信技术有限公司 | A device and method for secure information joint processing |
| US20110035781A1 (en) * | 2009-04-07 | 2011-02-10 | Pratyush Moghe | Distributed data search, audit and analytics |
| CN103152352A (en) * | 2013-03-15 | 2013-06-12 | 北京邮电大学 | Perfect information security and forensics monitoring method and system based on cloud computing environment |
| CN104426697A (en) * | 2013-08-29 | 2015-03-18 | 上海斐讯数据通信技术有限公司 | Network fault management system |
| CN103561018A (en) * | 2013-10-30 | 2014-02-05 | 蓝盾信息安全技术股份有限公司 | Intrusion detection real-time analysis system for big data application platform |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Spark-based log auditing and reversed checking system for big data platforms |
Non-Patent Citations (1)
| Title |
|---|
| 王帅等: "网络安全分析中的大数据技术应用", 《电信科学》 * |
Cited By (38)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106992872B (en) * | 2016-01-21 | 2020-05-12 | 中国移动通信集团公司 | Information processing method and system |
| CN106992872A (en) * | 2016-01-21 | 2017-07-28 | 中国移动通信集团公司 | A kind of method and system of information processing |
| CN105930373A (en) * | 2016-04-13 | 2016-09-07 | 北京思特奇信息技术股份有限公司 | Spark streaming based big data stream processing method and system |
| CN105915377A (en) * | 2016-04-14 | 2016-08-31 | 北京思特奇信息技术股份有限公司 | Flume and Spark streaming integration method and system |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
| CN107453882A (en) * | 2016-05-30 | 2017-12-08 | 北京京东尚科信息技术有限公司 | Warning information paradigmatic system and method in a kind of cluster |
| CN107453882B (en) * | 2016-05-30 | 2020-06-30 | 北京京东尚科信息技术有限公司 | Alarm information aggregation system and method in cluster |
| CN107547229A (en) * | 2016-06-29 | 2018-01-05 | 南京联成科技发展股份有限公司 | A kind of implementation method of the safe operation management platform intelligent control based on big data |
| CN105933186A (en) * | 2016-06-30 | 2016-09-07 | 北京奇虎科技有限公司 | Security detection method, device and system |
| CN107579944A (en) * | 2016-07-05 | 2018-01-12 | 南京联成科技发展股份有限公司 | Based on artificial intelligence and MapReduce security attack Forecasting Methodologies |
| CN107579944B (en) * | 2016-07-05 | 2020-08-11 | 南京联成科技发展股份有限公司 | Artificial intelligence and MapReduce-based security attack prediction method |
| CN106230819A (en) * | 2016-07-31 | 2016-12-14 | 上海交通大学 | A kind of DDoS detection method based on stream sampling |
| CN106230819B (en) * | 2016-07-31 | 2019-08-06 | 上海交通大学 | A kind of DDoS detection method based on stream sampling |
| CN106254125A (en) * | 2016-08-18 | 2016-12-21 | 南京联成科技发展有限公司 | The method and system of security incident correlation analysiss based on big data |
| CN106790572A (en) * | 2016-12-27 | 2017-05-31 | 广州华多网络科技有限公司 | The system and method that a kind of distributed information log is collected |
| CN107104951A (en) * | 2017-03-29 | 2017-08-29 | 国家电网公司 | The detection method and device of Attack Source |
| CN108964957B (en) * | 2017-05-24 | 2022-08-02 | 中兴通讯股份有限公司 | Method for monitoring data communication service quality and big data system |
| CN108964957A (en) * | 2017-05-24 | 2018-12-07 | 中兴通讯股份有限公司 | A kind of method and big data system of data communication service quality monitoring |
| CN107220348A (en) * | 2017-05-27 | 2017-09-29 | 郑州云海信息技术有限公司 | A kind of method of data capture based on Flume and Alluxio |
| CN108021809A (en) * | 2017-12-19 | 2018-05-11 | 北京明朝万达科技股份有限公司 | A kind of data processing method and system |
| CN108334646A (en) * | 2018-04-11 | 2018-07-27 | 焦点科技股份有限公司 | A kind of link structure optimization method based on frequent browsing sequence |
| CN108737549A (en) * | 2018-05-25 | 2018-11-02 | 江苏联盟信息工程有限公司 | A kind of log analysis method and device of big data quantity |
| CN108763562A (en) * | 2018-06-04 | 2018-11-06 | 广东京信软件科技有限公司 | A kind of construction method based on big data skill upgrading data exchange efficiency |
| CN110690984A (en) * | 2018-07-05 | 2020-01-14 | 上海宝信软件股份有限公司 | Spark-based big data weblog acquisition, analysis and early warning method and system |
| CN110968470A (en) * | 2018-09-28 | 2020-04-07 | 江苏赛融科技股份有限公司 | Operation and maintenance monitoring and aggregation management system |
| CN109525422A (en) * | 2018-10-31 | 2019-03-26 | 武汉雨滴科技有限公司 | A kind of daily record data method for managing and monitoring |
| CN109861844B (en) * | 2018-12-07 | 2021-09-03 | 中国人民大学 | Cloud service problem fine-grained intelligent tracing method based on logs |
| CN109861844A (en) * | 2018-12-07 | 2019-06-07 | 中国人民大学 | A kind of cloud service problem fine granularity intelligence source tracing method based on log |
| CN109951463A (en) * | 2019-03-07 | 2019-06-28 | 成都古河云科技有限公司 | A kind of Internet of Things big data analysis method stored based on stream calculation and novel column |
| CN110879771A (en) * | 2019-11-05 | 2020-03-13 | 北京航空航天大学 | Log analysis system for user anomaly detection based on keyword sequence mining |
| CN111131253A (en) * | 2019-12-24 | 2020-05-08 | 北京优特捷信息技术有限公司 | Scene-based security event global response method, device, equipment and storage medium |
| CN111209258A (en) * | 2019-12-31 | 2020-05-29 | 航天信息股份有限公司 | Tax end system log real-time analysis method, equipment, medium and system |
| CN111782477A (en) * | 2020-06-30 | 2020-10-16 | 平安国际智慧城市科技股份有限公司 | Abnormal log monitoring method and device, computer equipment and storage medium |
| CN111782477B (en) * | 2020-06-30 | 2023-02-14 | 深圳赛安特技术服务有限公司 | Abnormal log monitoring method and device, computer equipment and storage medium |
| CN113704069A (en) * | 2021-07-20 | 2021-11-26 | 北京直真科技股份有限公司 | Alarm system fault positioning method based on flash log collection technology |
| CN113542311A (en) * | 2021-09-17 | 2021-10-22 | 成都数默科技有限公司 | Method for detecting and backtracking defect host in real time |
| CN113542311B (en) * | 2021-09-17 | 2021-11-26 | 成都数默科技有限公司 | Method for detecting and backtracking defect host in real time |
| CN114760189A (en) * | 2022-03-30 | 2022-07-15 | 深信服科技股份有限公司 | Information determination method, equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105207826A (en) | Security attack alarm positioning system based on Spark big data platform of Tachyou | |
| US11831668B1 (en) | Using a logical graph to model activity in a network environment | |
| US10885393B1 (en) | Scalable incident-response and forensics toolkit | |
| US11095524B2 (en) | Component detection and management using relationships | |
| Khare et al. | Big data in IoT | |
| US10171565B2 (en) | Application monitoring for cloud-based architectures | |
| US10367827B2 (en) | Using network locations obtained from multiple threat lists to evaluate network data or machine data | |
| CN110650038B (en) | Security event log collecting and processing method and system for multiple classes of supervision objects | |
| EP3413512B1 (en) | Alarm information processing method, apparatus and system | |
| US20160359695A1 (en) | Network behavior data collection and analytics for anomaly detection | |
| CN107943668A (en) | Computer server cluster daily record monitoring method and monitor supervision platform | |
| US11770464B1 (en) | Monitoring communications in a containerized environment | |
| CN103152352A (en) | Perfect information security and forensics monitoring method and system based on cloud computing environment | |
| US9960975B1 (en) | Analyzing distributed datasets | |
| CN110138770B (en) | Threat information generation and sharing system and method based on Internet of things | |
| US11954130B1 (en) | Alerting based on pod communication-based logical graph | |
| CN111930886A (en) | Log processing method, system, storage medium and computer equipment | |
| CN108959445A (en) | Distributed information log processing method and processing device | |
| CN103856354A (en) | Method for achieving unified management of logs of cluster storage system | |
| CN103944763A (en) | Network-assistant management system and method of electrical power system | |
| CN105488191A (en) | Data acquisition processing method and device based on big data information safety management operation and maintenance service platform | |
| Laue et al. | A SIEM architecture for multidimensional anomaly detection | |
| CN107682166A (en) | The implementation method of safe O＆M service platform remote data acquisition based on big data | |
| CN116431324A (en) | Edge system based on Kafka high concurrency data acquisition and distribution | |
| Vega et al. | KISS methodologies for network management and anomaly detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 210012, Nanjing high tech Zone, Jiangsu, Nanjing Software Park, No. 99 unity Road, Eagle building, block A, 14 floor Applicant after: Nanjing Liancheng science and technology development Limited by Share Ltd Address before: A small road in Yuhuatai District of Nanjing City, Jiangsu province 210012 Building No. 158 Building 1 new ideal Applicant before: NANJING LIANCHENG TECHNOLOGY DEVELOPMENT CO., LTD. |
|
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 210000 14F, building A, Eagle building, 99 solidarity Road, Nanjing Software Park, Nanjing hi tech Zone, Jiangsu Applicant after: Nanjing Liancheng science and technology development Limited by Share Ltd Address before: 210012, Nanjing high tech Zone, Jiangsu, Nanjing Software Park, No. 99 unity Road, Eagle building, block A, 14 floor Applicant before: Nanjing Liancheng science and technology development Limited by Share Ltd |
|
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20151230 |
|
| RJ01 | Rejection of invention patent application after publication |