CN103384241A - Distributed analysis method and system for security event data - Google Patents
Distributed analysis method and system for security event data Download PDFInfo
- Publication number
- CN103384241A CN103384241A CN2012105606201A CN201210560620A CN103384241A CN 103384241 A CN103384241 A CN 103384241A CN 2012105606201 A CN2012105606201 A CN 2012105606201A CN 201210560620 A CN201210560620 A CN 201210560620A CN 103384241 A CN103384241 A CN 103384241A
- Authority
- CN
- China
- Prior art keywords
- security event
- event data
- analysis
- data
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 77
- 238000013500 data storage Methods 0.000 claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 8
- 238000007405 data analysis Methods 0.000 claims description 21
- 238000000034 method Methods 0.000 claims description 21
- 238000007726 management method Methods 0.000 claims description 12
- 238000007792 addition Methods 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 230000000737 periodic effect Effects 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 3
- 241001269238 Data Species 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009385 viral infection Effects 0.000 description 1
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed analysis method and system for security event data. Firstly, the security event data are received; a storage scheme is determined based on the magnitude of the received security event data and/or an existing hardware device and/or processing speed requirements, and security event data storage is conducted based on the storage scheme; the security event data which are needed are read and subjected to distributed analysis, and analysis results are summarized; a hardware device used for the security event data storage and a hardware device used for the distributed analysis are isolated from each other, and different hardware devices are used according to the requirements of storage and analysis respectively. Therefore, the analysis system can store massive security event data with complex structures, and the analysis speed is increased substantially because analysis and storage are separated.
Description
Technical field
The present invention relates to the computer network security technology field, relate in particular to a kind of distribution analysis method towards security event data and system.
Background technology
Along with the appearance of computer and universal, the information of magnanimity is present in network everywhere with digitized forms, science data, medical data, consensus data, finance data etc., and these data have also attracted a large amount of network attacks.And along with network attack is more and more, become increasingly complex, relevant security event data is of a great variety, and data structure is also and is complicated.For example distributed attack of DOS() event, it is attacked IP and is all uncertain by attacking the IP number, especially attacks IP, may reach thousands of; A virus infections event, nested a plurality of other behaviors of meeting in its behavior, this security event data usage data storehouse and data warehouse storage can not have been satisfied the demand.And, in time analyze and process a great problem that has also become network safety filed for the security event data of magnanimity.
Summary of the invention
For above-mentioned technical problem, the invention provides a kind of distribution analysis method towards security event data and system, carry out the security event data storage according to the storage scheme of formulating, and storage and analysis isolation mutually, configure hardware equipment as required separately, make storage and the analysis speed of security event data accelerate, can produce in time analysis result and feed back to external system.
The present invention adopts following method to realize: a kind of distribution analysis method towards security event data comprises:
Receive security event data;
Require to determine storage scheme based on the security event data magnitude that receives and/or existing hardware equipment and/or processing speed, and carry out security event data based on described storage scheme and store;
Read required security event data and carry out distributed analysis, and analysis result is gathered; Described distributed analysis refers to utilize a plurality of distributed nodes to complete the security event data analysis;
Be used for security event data storage and separated from one another for the hardware device of distributed analysis, adopt different hardware devices; Different task adopts different hardware devices, all reaches each other maximizing performance, and universal along with High_speed NIC, and the network data transmission between security event data storage and distributed analysis can not become bottleneck.
Also comprise in method: the User instruction is carried out additions and deletions management and/or timing management to relating in distributed analysis of task; Can realize the real-time, interactive of external system and analytical system.
Also comprise in method: monitor whole analytical system, as occur extremely reporting to the police.
Also comprise in method according to system configuration periodic cleaning analytical system; The expired daily record that produces when for example, moving for analytical system is processed automatically.
Adopt json form storage security event data in method; Can satisfy the description demand of the complex datas such as array, subobject.
Adopt Map/Reduce to carry out distributed analysis in method; Utilization is implemented distributed analysis based on the high speed Map/Reduce of internal memory, carries out data mining, resolves task result, and the task result of resolving is returned to the outside.
The hardware device that is used for the security event data storage described in method and is used for distributed analysis can carry out the distributed linear expansion; Namely when the hardware device that is used for the security event data storage or is used for distributed analysis becomes the performance bottleneck of analytical system, corresponding hardware device is implemented separately the distributed linear expansion; For example, the storage line server is adopted in expansion security event data storage, can adopt the calculation type server of the large internal memory of many CPU when expanding the hardware device that distributed analysis uses.
Storage scheme described in method can comprise: a plurality of backups of identical data or the storage of same data multimachine burst; Namely when certain server that is used for the storage security event data machine of delaying, can in time enable the related data of storing in backup server, in order to reach certain analysis speed requirement, same security event data burst can be stored in different servers, storage speed can be significantly improved like this.
A kind of distributed analysis system towards security event data comprises:
Data pour into module, receive security event data, deposit described security event data in data memory module;
Data memory module based on the security event data magnitude that receives and/or existing hardware equipment and/or the definite storage scheme of processing speed requirement, and carries out the security event data storage based on described storage scheme;
Data analysis module reads required security event data and carries out distributed analysis from data memory module, and analysis result is gathered; Described distributed analysis refers to utilize a plurality of distributed nodes to complete the security event data analysis;
Described data memory module and data analysis module are separated from one another, adopt different hardware devices; All reach each other maximizing performance, and universal along with High_speed NIC, the network data transmission between security event data storage and distributed analysis can not become bottleneck.
Also comprise in system: the data interaction module, the User instruction is carried out additions and deletions management and/or timing management to relating in data analysis module of task; Can realize the real-time, interactive of external system and analytical system.
Also comprise in system: the system monitoring module is used for monitoring whole analytical system, as occurs extremely reporting to the police.
Also comprise in system: the cleaned system module, according to system configuration periodic cleaning analytical system; The expired daily record that produces when for example, moving for analytical system is processed automatically.
Data memory module described in system adopts json form storage security event data; Can satisfy the description demand of the complex datas such as array, subobject.
Data analysis module described in system adopts Map/Reduce to carry out distributed analysis; Utilization is implemented distributed analysis based on the high speed Map/Reduce of internal memory, carries out data mining, resolves task result, and the task result of resolving is returned to the outside.
The hardware device of data memory module described in system and data analysis module can carry out the distributed linear expansion; Namely when the hardware device that is used for the security event data storage or is used for distributed analysis becomes the performance bottleneck of analytical system, corresponding hardware device is implemented separately the distributed linear expansion; For example, the storage line server is adopted in expansion security event data storage, can adopt the calculation type server of the large internal memory of many CPU when expanding the hardware device that distributed analysis uses.
Described in system, storage scheme comprises: a plurality of backups of identical data or the storage of same data multimachine burst; Namely when certain server that is used for the storage security event data machine of delaying, can in time enable the related data of storing in backup server, in order to reach certain analysis speed requirement, same security event data burst can be stored in different servers, storage speed has very significantly and improves like this.
In sum, the invention provides a kind of distribution analysis method towards security event data and system, at first, based on security event data magnitude and/or existing hardware equipment and/or processing speed requirement deployment storage scheme, can a plurality of backups of identical data or the storage of same data fragmentation, and will store from analysis portion and be deployed on different hardware devices.Improved like this storage and the analysis speed of security event data, become simply for storage and the analysis of the security event data of relative complex, easily; And storage and analysis can be carried out linear expansion and improvement as required separately, make analyzing system performance reach optimum.
Description of drawings
In order to be illustrated more clearly in technical scheme of the present invention, the below will do to introduce simply to the accompanying drawing of required use in embodiment, apparently, the accompanying drawing that the following describes is only some embodiment that put down in writing in the present invention, for those of ordinary skills, under the prerequisite of not paying creative work, can also obtain according to these accompanying drawings other accompanying drawing.
Fig. 1 is a kind of distribution analysis method flow chart towards security event data provided by the invention;
Fig. 2 is a kind of distributed analysis system structure chart towards security event data provided by the invention.
Embodiment
The present invention has provided a kind of distribution analysis method towards security event data and system, in order to make those skilled in the art person understand better technical scheme in the embodiment of the present invention, and above-mentioned purpose of the present invention, feature and advantage can be become apparent more, below in conjunction with accompanying drawing, technical scheme in the present invention is described in further detail:
At first the present invention provides a kind of distribution analysis method towards security event data, as shown in Figure 1, comprising:
S101 receives security event data;
S102 requires to determine storage scheme based on the security event data magnitude that receives and/or existing hardware equipment and/or processing speed, and carries out security event data based on described storage scheme and store;
S103 reads required security event data and carries out distributed analysis, and analysis result is gathered; Described distributed analysis refers to utilize a plurality of distributed nodes to complete the security event data analysis;
Be used for security event data storage and separated from one another for the hardware device of distributed analysis, adopt different hardware devices.
Preferably, described method also comprises: the User instruction is carried out additions and deletions management and/or timing management to relating in distributed analysis of task.
Preferably, described method also comprises: monitor whole analytical system, as occur extremely reporting to the police.
Preferably, described method also comprises: according to system configuration periodic cleaning analytical system.
Preferably, described method adopts json form storage security event data.
Preferably, described method adopts Map/Reduce to carry out distributed analysis.
Preferably, describedly can carry out the distributed linear expansion for security event data storage and the hardware device that is used for distributed analysis.
Preferably, described storage scheme comprises: a plurality of backups of identical data or the storage of same data multimachine burst.
The present invention also provides a kind of distributed analysis system towards security event data, as shown in Figure 2, comprising:
Data pour into module 201, receive security event data, deposit described security event data in data memory module 202;
Described data memory module 202 is separated from one another with data analysis module 203, adopts different hardware devices.
Preferably, also comprise in system: the data interaction module, the User instruction is carried out additions and deletions management and/or timing management to relating in data analysis module of task.
Preferably, also comprise in system: the system monitoring module is used for monitoring whole analytical system, as occurs extremely reporting to the police.
Preferably, also comprise in system: the cleaned system module, according to system configuration periodic cleaning analytical system.
Preferably, data memory module described in system adopts json form storage security event data.
Preferably, described data analysis module adopts Map/Reduce to carry out distributed analysis.
Preferably, the hardware device of described data memory module and data analysis module can carry out the distributed linear expansion as required.
Preferably, described storage scheme comprises: a plurality of backups of identical data or the storage of same data multimachine burst.
as mentioned above, the present invention has provided a kind of distribution analysis method towards security event data and system, the difference of itself and conventional method is, for conventional method, the security event data storage is carried out on a server substantially with analysis, limited like this analysis speed, the present invention separates the security event data storage and carries out with analysis, can select as required separately optimum hardware configuration scheme, and storage can be formulated storage scheme as required for security event data, therefore, analysis result can be stored and in time produce to security event data for the magnanimity complexity fast, return to external system.
Above embodiment is unrestricted technical scheme of the present invention in order to explanation.Any modification or partial replacement that does not break away from spirit and scope of the invention all should be encompassed in the middle of claim scope of the present invention.
Claims (16)
1. the distribution analysis method towards security event data, is characterized in that, comprising:
Receive security event data;
Require to determine storage scheme based on the security event data magnitude that receives and/or existing hardware equipment and/or processing speed, and carry out security event data based on described storage scheme and store;
Read required security event data and carry out distributed analysis, and analysis result is gathered; Described distributed analysis refers to utilize a plurality of distributed nodes to complete the security event data analysis;
Be used for security event data storage and separated from one another for the hardware device of distributed analysis, adopt different hardware devices.
2. the method for claim 1, is characterized in that, also comprises: the User instruction is carried out additions and deletions management and/or timing management to relating in distributed analysis of task.
3. the method for claim 1, is characterized in that, also comprises: monitor whole analytical system, as occur extremely reporting to the police.
4. the method for claim 1, is characterized in that, also comprises: according to system configuration periodic cleaning analytical system.
5. the method for claim 1, is characterized in that, adopts json form storage security event data.
6. the method for claim 1, is characterized in that, adopts Map/Reduce to carry out distributed analysis.
7. the method for claim 1, is characterized in that, the described storage with the hardware device that is used for distributed analysis for security event data can be carried out the distributed linear expansion.
8. the method for claim 1, is characterized in that, described storage scheme comprises: a plurality of backups of identical data or the storage of same data multimachine burst.
9. the distributed analysis system towards security event data, is characterized in that, comprising:
Data pour into module, receive security event data, deposit described security event data in data memory module;
Data memory module based on the security event data magnitude that receives and/or existing hardware equipment and/or the definite storage scheme of processing speed requirement, and carries out the security event data storage based on described storage scheme;
Data analysis module reads required security event data and carries out distributed analysis from data memory module, and analysis result is gathered; Described distributed analysis refers to utilize a plurality of distributed nodes to complete the security event data analysis;
Described data memory module and data analysis module are separated from one another, adopt different hardware devices.
10. system as claimed in claim 9, is characterized in that, also comprises: the data interaction module, the User instruction is carried out additions and deletions management and/or timing management to relating in data analysis module of task.
11. system as claimed in claim 9 is characterized in that, also comprises: the system monitoring module is used for monitoring whole analytical system, as occurs extremely reporting to the police.
12. system as claimed in claim 9 is characterized in that, also comprises: the cleaned system module, according to system configuration periodic cleaning analytical system.
13. system as claimed in claim 9 is characterized in that, described data memory module adopts json form storage security event data.
14. system as claimed in claim 9 is characterized in that, described data analysis module adopts Map/Reduce to carry out distributed analysis.
15. system as claimed in claim 9 is characterized in that, the hardware device of described data memory module and data analysis module can carry out the distributed linear expansion.
16. system as claimed in claim 9 is characterized in that, described storage scheme comprises: a plurality of backups of identical data or the storage of same data multimachine burst.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210560620.1A CN103384241B (en) | 2012-12-21 | 2012-12-21 | A kind of distribution analysis method towards security event data and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210560620.1A CN103384241B (en) | 2012-12-21 | 2012-12-21 | A kind of distribution analysis method towards security event data and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103384241A true CN103384241A (en) | 2013-11-06 |
| CN103384241B CN103384241B (en) | 2016-07-13 |
Family
ID=49491938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210560620.1A Active CN103384241B (en) | 2012-12-21 | 2012-12-21 | A kind of distribution analysis method towards security event data and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103384241B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1949720A (en) * | 2006-09-08 | 2007-04-18 | 中山大学 | Distributed network invasion detecting system |
| CN101572691A (en) * | 2008-04-30 | 2009-11-04 | 华为技术有限公司 | Method, system and device for intrusion detection |
| CN101582883A (en) * | 2009-06-26 | 2009-11-18 | 西安电子科技大学 | System and method for managing security of general network |
| CN101714958A (en) * | 2009-10-31 | 2010-05-26 | 福建伊时代信息科技股份有限公司 | Multifunctional comprehensive security gateway system |
| CN102131224A (en) * | 2010-07-26 | 2011-07-20 | 北京创和世纪通讯技术有限公司 | Wireless network evaluation system and method |
-
2012
- 2012-12-21 CN CN201210560620.1A patent/CN103384241B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1949720A (en) * | 2006-09-08 | 2007-04-18 | 中山大学 | Distributed network invasion detecting system |
| CN101572691A (en) * | 2008-04-30 | 2009-11-04 | 华为技术有限公司 | Method, system and device for intrusion detection |
| CN101582883A (en) * | 2009-06-26 | 2009-11-18 | 西安电子科技大学 | System and method for managing security of general network |
| CN101714958A (en) * | 2009-10-31 | 2010-05-26 | 福建伊时代信息科技股份有限公司 | Multifunctional comprehensive security gateway system |
| CN102131224A (en) * | 2010-07-26 | 2011-07-20 | 北京创和世纪通讯技术有限公司 | Wireless network evaluation system and method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103384241B (en) | 2016-07-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11310313B2 (en) | Multi-threaded processing of search responses returned by search peers | |
| US11184467B2 (en) | Multi-thread processing of messages | |
| US9344443B1 (en) | Finding command and control center computers by communication link tracking | |
| CN105630847B (en) | Date storage method, data query method, apparatus and system | |
| CN106487596A (en) | Distributed Services follow the tracks of implementation method | |
| CN104731956A (en) | Method and system for synchronizing data and related database | |
| US11848940B2 (en) | Cumulative trajectory of cyber reconnaissance indicators | |
| CN103324713B (en) | Data processing method and device in multistage server and data processing system | |
| WO2013032909A1 (en) | Multidimension column-based partitioning and storage | |
| KR101679573B1 (en) | Method and apparatus for service traffic security using dimm channel distribution multicore processing system | |
| CN104618304A (en) | Data processing method and data processing system | |
| Las-Casas et al. | A big data architecture for security data and its application to phishing characterization | |
| CN114003904B (en) | Information sharing method, device, computer equipment and storage medium | |
| CN106599120A (en) | Stream processing framework-based data processing method and apparatus | |
| CN105159820A (en) | Transmission method and device of system log data | |
| CN106936917A (en) | A kind of cloud storage method and device, electronic equipment | |
| CN108351940B (en) | System and method for high frequency heuristic data acquisition and analysis of information security events | |
| CN103384241A (en) | Distributed analysis method and system for security event data | |
| Kang et al. | Large scale complex network analysis using the hybrid combination of a MapReduce cluster and a highly multithreaded system | |
| CN104618219A (en) | Cross-machine-room data synchronizing method and device | |
| CN204360302U (en) | A kind of information management cloud stores all-in-one | |
| CN106412915A (en) | Pseudo-wireless access point identification method and system | |
| US10362062B1 (en) | System and method for evaluating security entities in a computing environment | |
| CN115174594B (en) | Data synchronization method, device, equipment and medium of distributed system | |
| CN105574539B (en) | A kind of DNS log analysis methods and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 100080 Beijing city Haidian District minzhuang Road No. 3, Tsinghua Science Park Building 1 Yuquan Huigu a Patentee after: Beijing ahtech network Safe Technology Ltd Address before: 100080 Haidian District City, Zhongguancun, the main street, No. 1 Hailong building, room 1415, room 14 Patentee before: Beijing Antiy Electronic Installation Co., Ltd. |
|
| CP03 | Change of name, title or address | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Distributed analysis method and system for security event data Effective date of registration: 20170821 Granted publication date: 20160713 Pledgee: CITIC Bank Harbin branch Pledgor: Beijing ahtech network Safe Technology Ltd Registration number: 2017990000776 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20180817 Granted publication date: 20160713 Pledgee: CITIC Bank Harbin branch Pledgor: Beijing ahtech network Safe Technology Ltd Registration number: 2017990000776 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right | ||
| PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: Distributed analysis method and system for security event data Effective date of registration: 20180817 Granted publication date: 20160713 Pledgee: CITIC Bank Harbin branch Pledgor: Beijing ahtech network Safe Technology Ltd Registration number: 2018990000700 |
|
| PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
| PC01 | Cancellation of the registration of the contract for pledge of patent right |
Date of cancellation: 20191021 Granted publication date: 20160713 Pledgee: CITIC Bank Harbin branch Pledgor: Beijing ahtech network Safe Technology Ltd Registration number: 2018990000700 |
|
| PC01 | Cancellation of the registration of the contract for pledge of patent right |