CN111814142A - Big data rapid threat detection system based on OpenIOC - Google Patents
Big data rapid threat detection system based on OpenIOC Download PDFInfo
- Publication number
- CN111814142A CN111814142A CN202010601122.1A CN202010601122A CN111814142A CN 111814142 A CN111814142 A CN 111814142A CN 202010601122 A CN202010601122 A CN 202010601122A CN 111814142 A CN111814142 A CN 111814142A
- Authority
- CN
- China
- Prior art keywords
- open
- data
- ioc
- big data
- detection system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 21
- 238000006243 chemical reaction Methods 0.000 claims abstract description 4
- 238000011897 real-time detection Methods 0.000 claims abstract description 4
- 238000010606 normalization Methods 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 6
- 238000011160 research Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 230000003993 interaction Effects 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 238000003066 decision tree Methods 0.000 abstract description 6
- 238000010801 machine learning Methods 0.000 abstract description 4
- 238000004458 analytical method Methods 0.000 description 6
- 235000010627 Phaseolus vulgaris Nutrition 0.000 description 4
- 244000046052 Phaseolus vulgaris Species 0.000 description 4
- 238000013480 data collection Methods 0.000 description 4
- 230000006399 behavior Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000009545 invasion Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/554—Detecting local intrusion or implementing counter-measures involving event detection and direct action
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2246—Trees, e.g. B+trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24561—Intermediate data storage techniques for performance improvement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Computational Linguistics (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a big data rapid threat detection system based on Open IOC, which utilizes beacons to collect IT environment full stack event data, uses Elastic Search to realize the storage and retrieval of PB level data, and realizes the quasi-real-time detection of the Open IOC strategy in a threat library to the full stack event data based on an Open IOC retrieval conversion interface. The system predicts the branch condition retrieval speed by using a machine learning algorithm in combination with the data category and the data volume, intelligently adjusts the retrieval sequence of the decision tree, greatly improves the Open IOC strategy detection speed in a big data environment, and provides powerful support for timely discovering the IT system threat and disposing.
Description
Technical Field
The invention relates to the technical field of computer information processing, in particular to an Open IOC-based big data rapid threat detection system.
Background
At present, the scale and complexity of information systems are increasing, and these complex systems, applications and their security defense facilities continuously generate a large amount of information during operation. The information records various events occurring in the IT environment system every day, and the reason of the fault can be checked by monitoring and analyzing a large amount of fine information, the using behaviors of the user can be monitored, and abnormal conditions or traces left by an attacker when the user is attacked can be found. With the deep research on threat intelligence and the combination of more and more complex IT environments of users, threat indexes represented by Open IOCs greatly expand a complex threat event description index item set, and meanwhile, the data volume needing to be detected is increased by orders of magnitude, and the traditional Open IOC detection mechanism based on a host machine is not suitable for the requirement of the current IT environment on security threat detection.
Open Indicator of compliance (Open threat Indicator) is an intelligence sharing specification promulgated by MANDIANT corporation (later purchased by fireeye). Open IOC is a format for recording, defining, and sharing threat intelligence that enables rapid sharing of different types of threat intelligence by means of a machine-readable form. The IOC (indicator of evidence) is a technical indicator which is defined by MANDIANT in long-term digital evidence practice and can reflect the host or network behavior, and the IOC describes the event response information capturing various threats in an XML document type, including the attribute of virus files, the characteristic of registry change, virtual memory and the like, and is an index which can be proved after invasion and can identify one host or the whole network. By following the standard, logical groupings of IOCs can be established to communicate in a machine in a readable format to enable communication sharing of threat intelligence. For example, an event response team may write multiple IOCs using the Open IOC specification to describe the technical commonality of a threat. The native Open IOC tool's depiction of behavior is implemented by a combination of metrics that are limited to the host and the network.
Disclosure of Invention
The invention aims to provide an Open IOC-based big data rapid threat detection system, and realizes a system for rapidly detecting the whole IT environment by using Open IOC threat information under a big data environment.
In order to achieve the purpose, the invention provides the following technical scheme:
a big data rapid threat detection system based on Open IOC utilizes beacons to collect IT environment full stack event data, Elastic Search is used to realize storage and retrieval of data up to PB level, and an Open IOC retrieval conversion interface is used to realize quasi real-time detection of Open IOC strategy in a threat library on the full stack event data.
Preferably, the event data collection module is responsible for collecting full stack event information in the IT environment, including but not limited to physical information of a host, a server, a storage device, an exchange device, and a security device, and various index information and logs of software such as an operating system, a web server, middleware, an application system, and the like, and host traffic information. In order to cover the full stack data, the module supports the collection of a client and a standard protocol, and the client mainly collects the relevant event data of a host and a server based on native beans and custom beans; various equipment event information is collected through standard protocols such as syslog, SNMP and the like; meanwhile, the event data collection module is also responsible for protocol analysis of the flow.
The tables is a platform constructed by using Golang, and the libbaby is a core library of the tables, is used for providing an API (application programming interface) for connecting with a center or a cloud, and can also configure input characteristics and realize information collection and other works. An output module (Publisher) is encapsulated therein, and the output module can be responsible for sending collected data to a center or a cloud. Because go language is designed with a channel, the logic code for collecting data and the Publisher are communicated through the channel, and the coupling degree is lowest. Therefore, a collector is developed, the existence of the Publisher is not required to be known at all, and the data is sent to the server side when the program runs.
Preferably, the transmission and normalization module aggregates data from the data acquisition module in an asynchronous production/consumption manner, and the aggregation and normalization is realized through open source message middleware such as Kafka, socket MQ or Active MQ. Meanwhile, the module analyzes the event data, structures the event data, realizes attribute labeling such as classification and grade of the event data, and realizes event normalization, which is also a necessary link for matching the Open IOC rule with the event.
Preferably, the Open IOC threat library is responsible for managing self-research or externally introduced strategies that meet the Open IOC standard, and the strategies are structured after being imported into the threat library, so as to provide basic data for subsequent branch condition decomposition.
Preferably, the Open IOC big data retrieval module establishes an index by using an Elastic Search to store log records; the logical description structure of Open IOCs is converted into a big data retrieval language so that each Open IOC rule can traverse all the categories of events IT involves in the IT environment. The module predicts the possibility of retrieval hit and retrieval speed in each branch condition of the complex IOC through a machine learning algorithm, simultaneously converts the IOC branch condition into a decision tree, and selects the optimal prediction speed to traverse the decision tree according to the learning result, so as to realize the rapid matching of the Open IOC rule in a big data environment.
Elastic Search is a scalable open source full text Search and analysis engine. It can quickly store, search and analyze mass data. The Elastic Search is constructed based on mature Apache Lucene, is generated for large data during design, and can easily perform large-scale transverse expansion to support the processing of PB-level structured and unstructured mass data. The Elastic Search ecosphere has a good development state, and integrates a plurality of peripheral auxiliary systems, such as Marvel monitoring, Logstash analysis, and safety Shield.
Preferably, the Web module is responsible for foreground human-computer interaction and background management, and the foreground realizes a multi-dimensional query function through a Web page; the background provides the functions of strategy maintenance, big data cluster management, client management, user management and authority management.
Compared with the prior art, the invention has the beneficial effects that:
the Open IOC-based big data rapid threat detection system realizes the collection and centralized storage of IT environment events, realizes the detection of Open IOC rules on complete IT environment events, breaks the limitation that a native Open IOC stand-alone tool can only detect host information, and simultaneously avoids the problem that stand-alone version manual synchronization is not timely; the system predicts the branch condition retrieval speed by using a machine learning algorithm in combination with the data category and the data volume, intelligently adjusts the retrieval sequence of the decision tree, greatly improves the Open IOC strategy detection speed in a big data environment, and provides powerful support for timely discovering the IT system threat and disposing.
Drawings
FIG. 1 is a block flow diagram of an Open IOC-based big data rapid threat detection system according to the present invention;
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A big data rapid threat detection system based on Open IOC utilizes beacons to collect IT environment full stack event data, Elastic Search is used to realize storage and retrieval of data up to PB level, and an Open IOC retrieval conversion interface is used to realize quasi real-time detection of Open IOC strategy in a threat library on the full stack event data.
Furthermore, the event data collection module is responsible for collecting full stack event information in the IT environment, including but not limited to physical information of hosts, servers, storage devices, switching devices, and security devices, and various index information and logs of software such as operating systems, web servers, middleware, and application systems, and host traffic information. In order to cover the full stack data, the module supports the collection of a client and a standard protocol, and the client mainly collects the relevant event data of a host and a server based on native beans and custom beans; various equipment event information is collected through standard protocols such as syslog, SNMP and the like; meanwhile, the event data collection module is also responsible for protocol analysis of the flow.
Furthermore, the beats is a platform constructed by using Golang, and the libbeat is a core library of the platform, is used for providing an API (application programming interface) to connect with a center or a cloud, and can also configure input characteristics and realize information collection and other works. An output module (Publisher) is encapsulated therein, and the output module can be responsible for sending collected data to a center or a cloud. Because go language is designed with a channel, the logic code for collecting data and the Publisher are communicated through the channel, and the coupling degree is lowest. Therefore, a collector is developed, the existence of the Publisher is not required to be known at all, and the data is sent to the server side when the program runs.
Further, the transmission and normalization module aggregates data from the data acquisition module in an asynchronous production/consumption mode, and the data is realized through open source message middleware such as Kafka, Rocket MQ or Active MQ. Meanwhile, the module analyzes the event data, structures the event data, realizes attribute labeling such as classification and grade of the event data, and realizes event normalization, which is also a necessary link for matching the Open IOC rule with the event.
Furthermore, the Open IOC threat library is responsible for managing self-research or externally introduced strategies which accord with the Open IOC standard, and the strategies are structured after being imported into the threat library, so that basic data are provided for subsequent branch condition decomposition.
Further, the Open IOC big data retrieval module establishes an index by using an Elastic Search to store log records; the logical description structure of Open IOCs is converted into a big data retrieval language so that each Open IOC rule can traverse all the categories of events IT involves in the IT environment. The module predicts the possibility of retrieval hit and retrieval speed in each branch condition of the complex IOC through a machine learning algorithm, simultaneously converts the IOC branch condition into a decision tree, and selects the optimal prediction speed to traverse the decision tree according to the learning result, so as to realize the rapid matching of the Open IOC rule in a big data environment.
Further, Elastic Search is a scalable open source full text Search and analysis engine. It can quickly store, search and analyze mass data. The Elastic Search is constructed based on mature Apache Lucene, is generated for large data during design, and can easily perform large-scale transverse expansion to support the processing of PB-level structured and unstructured mass data. The Elastic Search ecosphere has a good development state, and integrates a plurality of peripheral auxiliary systems, such as Marvel monitoring, Logstash analysis, and safety Shield.
Furthermore, the Web module is responsible for foreground human-computer interaction and background management, and the foreground realizes a multi-dimensional query function through a Web page; the background provides the functions of strategy maintenance, big data cluster management, client management, user management and authority management.
Furthermore, the event data acquisition module is not limited to the client and the standard protocol, and the data existing in the database and the data provided by the user system through the API can also be used as a data source.
Further, the message queue in the transmission and normalization module is taken as an optional item, and mainly faces an environment with large transmission data volume or unstable transmission.
Further, the big data retrieval base platform may use solr.
Further, the web module does not restrict the front end framework.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (6)
1. An Open IOC-based big data rapid threat detection system is characterized in that: the detection system utilizes beacons to acquire IT environment full stack event data, uses Elastic Search to realize storage and retrieval of data up to PB level, and based on an Open IOC retrieval conversion interface, realizes quasi-real-time detection of Open IOC strategies in a threat library on the full stack event data.
2. The Open IOC-based big data rapid threat detection system of claim 1, wherein: the event data acquisition module is responsible for acquiring full stack event information in the IT environment, including but not limited to physical information of a host, a server, a storage device, an exchange device and a safety device, and various index information of software such as an operating system, a web server, middleware, an application system and the like, logs and host flow information.
3. The Open IOC-based big data rapid threat detection system of claim 1, wherein: the transmission and normalization module is used for summarizing data from the data acquisition module in a production/consumption asynchronous mode and is realized through open source message middleware such as Kafka, Rocket MQ or Active MQ.
4. The Open IOC-based big data rapid threat detection system of claim 1, wherein: the Open IOC threat library is responsible for managing self-research or externally introduced strategies which accord with the Open IOC standard, and the strategies are structured after being introduced into the threat library, so that basic data are provided for subsequent branch condition decomposition.
5. The Open IOC-based big data rapid threat detection system of claim 1, wherein: the OpenIOC big data retrieval module is used for storing log records by using Elastic Search and establishing indexes; the logical description structure of Open IOCs is converted into a big data retrieval language so that each OpenIOC rule can traverse all the categories of events IT involves in the IT environment.
6. The Open IOC-based big data rapid threat detection system of claim 1, wherein: the Web module is responsible for foreground human-computer interaction and background management, and the foreground realizes a multi-dimensional query function through a Web page; the background provides the functions of strategy maintenance, big data cluster management, client management, user management and authority management.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010601122.1A CN111814142A (en) | 2020-06-29 | 2020-06-29 | Big data rapid threat detection system based on OpenIOC |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010601122.1A CN111814142A (en) | 2020-06-29 | 2020-06-29 | Big data rapid threat detection system based on OpenIOC |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111814142A true CN111814142A (en) | 2020-10-23 |
Family
ID=72855896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010601122.1A Pending CN111814142A (en) | 2020-06-29 | 2020-06-29 | Big data rapid threat detection system based on OpenIOC |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111814142A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106778253A (en) * | 2016-11-24 | 2017-05-31 | 国家电网公司 | Threat context aware information security Initiative Defense model based on big data |
| CN107196910A (en) * | 2017-04-18 | 2017-09-22 | 国网山东省电力公司电力科学研究院 | Threat early warning monitoring system, method and the deployment framework analyzed based on big data |
| CN108460278A (en) * | 2018-02-13 | 2018-08-28 | 北京奇安信科技有限公司 | A kind of threat information processing method and device |
| CN110719291A (en) * | 2019-10-16 | 2020-01-21 | 杭州安恒信息技术股份有限公司 | Network threat identification method and identification system based on threat information |
| CN110825792A (en) * | 2019-11-15 | 2020-02-21 | 珠海市新德汇信息技术有限公司 | High-concurrency distributed data retrieval method based on golang middleware coroutine mode |
| CN110955893A (en) * | 2019-11-22 | 2020-04-03 | 杭州安恒信息技术股份有限公司 | Malicious file threat analysis platform and malicious file threat analysis method |
-
2020
- 2020-06-29 CN CN202010601122.1A patent/CN111814142A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106778253A (en) * | 2016-11-24 | 2017-05-31 | 国家电网公司 | Threat context aware information security Initiative Defense model based on big data |
| CN107196910A (en) * | 2017-04-18 | 2017-09-22 | 国网山东省电力公司电力科学研究院 | Threat early warning monitoring system, method and the deployment framework analyzed based on big data |
| CN108460278A (en) * | 2018-02-13 | 2018-08-28 | 北京奇安信科技有限公司 | A kind of threat information processing method and device |
| CN110719291A (en) * | 2019-10-16 | 2020-01-21 | 杭州安恒信息技术股份有限公司 | Network threat identification method and identification system based on threat information |
| CN110825792A (en) * | 2019-11-15 | 2020-02-21 | 珠海市新德汇信息技术有限公司 | High-concurrency distributed data retrieval method based on golang middleware coroutine mode |
| CN110955893A (en) * | 2019-11-22 | 2020-04-03 | 杭州安恒信息技术股份有限公司 | Malicious file threat analysis platform and malicious file threat analysis method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111984499B (en) | Fault detection method and device for big data cluster | |
| CN111190876A (en) | Log management system and operation method thereof | |
| CN111885012B (en) | Network situation perception method and system based on information acquisition of various network devices | |
| CN108763957B (en) | Database security audit system, method and server | |
| CN110740141A (en) | integration network security situation perception method, device and computer equipment | |
| EP4099170B1 (en) | Method and apparatus of auditing log, electronic device, and medium | |
| CN108197261A (en) | A kind of wisdom traffic operating system | |
| CN106371986A (en) | Log treatment operation and maintenance monitoring system | |
| CN106815125A (en) | A kind of log audit method and platform | |
| CN114219374B (en) | Big data analysis decision system and method based on block chain | |
| CN114548706A (en) | Early warning method for business risk and related equipment | |
| CN112416872A (en) | Cloud platform log management system based on big data | |
| CN112306820B (en) | Log operation and maintenance root cause analysis method and device, electronic equipment and storage medium | |
| CN111740868A (en) | Alarm data processing method and device and storage medium | |
| CN108390782A (en) | A kind of centralization application system performance question synthesis analysis method | |
| KR20220166760A (en) | Apparatus and method for managing trouble using big data of 5G distributed cloud system | |
| CN116257021A (en) | Intelligent network security situation monitoring and early warning platform for industrial control system | |
| CN116961241B (en) | Unified application monitoring platform based on power grid business | |
| Zou et al. | Improving log-based fault diagnosis by log classification | |
| CN111814142A (en) | Big data rapid threat detection system based on OpenIOC | |
| CN114238027A (en) | Multi-dimensional analysis system based on mass request data | |
| Polozhentsev et al. | Novel Cyber Incident Management System for 5G-based Critical Infrastructures | |
| CN117640432B (en) | Operation and maintenance monitoring method for distributed data center | |
| US11835989B1 (en) | FPGA search in a cloud compute node | |
| CN114422324B (en) | Alarm information processing method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |