CN117640142A - Full-process security operation threat discovery system and method - Google Patents
Full-process security operation threat discovery system and method Download PDFInfo
- Publication number
- CN117640142A CN117640142A CN202311380230.0A CN202311380230A CN117640142A CN 117640142 A CN117640142 A CN 117640142A CN 202311380230 A CN202311380230 A CN 202311380230A CN 117640142 A CN117640142 A CN 117640142A
- Authority
- CN
- China
- Prior art keywords
- data
- threat
- intelligence
- information
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000004458 analytical method Methods 0.000 claims abstract description 104
- 238000003860 storage Methods 0.000 claims abstract description 56
- 238000001514 detection method Methods 0.000 claims abstract description 25
- 238000007781 pre-processing Methods 0.000 claims abstract description 13
- 238000010921 in-depth analysis Methods 0.000 claims abstract description 5
- 230000008520 organization Effects 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 25
- 230000008569 process Effects 0.000 claims description 20
- 238000005065 mining Methods 0.000 claims description 17
- 238000004590 computer program Methods 0.000 claims description 16
- 230000035945 sensitivity Effects 0.000 claims description 15
- 238000011835 investigation Methods 0.000 claims description 14
- 230000004927 fusion Effects 0.000 claims description 12
- 238000010219 correlation analysis Methods 0.000 claims description 8
- 238000011084 recovery Methods 0.000 claims description 7
- 238000012550 audit Methods 0.000 claims description 6
- 230000002085 persistent effect Effects 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 230000008439 repair process Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 2
- 238000012217 deletion Methods 0.000 claims 1
- 230000037430 deletion Effects 0.000 claims 1
- 238000012098 association analyses Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 15
- 238000007726 management method Methods 0.000 description 13
- 238000012544 monitoring process Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 5
- 238000011160 research Methods 0.000 description 5
- 230000003068 static effect Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000010365 information processing Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000007418 data mining Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000007123 defense Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 108090000623 proteins and genes Proteins 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000001066 destructive effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000013439 planning Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/30—Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information
- H04L63/302—Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information gathering intelligence information for situation awareness or reconnaissance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0263—Rule management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1433—Vulnerability analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Evolutionary Computation (AREA)
- Technology Law (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
本发明提供一种全流程安全运营威胁发现系统及方法，系统包括：数据接入层、存储层、工具层和分析层；通过数据接入层采集多源情报数据，并对多源情报数据进行预处理后存入存储层；存储层将多源情报数据进行分类存储管理，构建结构化数据仓库与非结构化数据湖；在工具层基于工具集对多源情报数据进行深度挖掘与融合，生成情报信息报告与情报产品；在分析层基于情报信息报告及情报产品，对外部威胁进行深入分析，生成威胁评估与预警信息；通过上述系统，可以准确感知安全态势，进一步提高了检测和威胁发现的有效性和高效性。
The invention provides a full-process security operation threat discovery system and method. The system includes: a data access layer, a storage layer, a tool layer and an analysis layer; multi-source intelligence data is collected through the data access layer, and the multi-source intelligence data is processed After preprocessing, it is stored in the storage layer; the storage layer classifies, stores and manages multi-source intelligence data, and builds structured data warehouses and unstructured data lakes; at the tool layer, the multi-source intelligence data is deeply mined and integrated based on tool sets to generate Intelligence information reports and intelligence products; at the analysis layer, based on intelligence information reports and intelligence products, we conduct in-depth analysis of external threats and generate threat assessment and early warning information; through the above system, we can accurately perceive the security situation, further improving the efficiency of detection and threat discovery. Effectiveness and efficiency.
Description
Technical Field
The invention relates to the technical field of network security, in particular to a full-flow security operation threat discovery system and method.
Background
With the rapid development of the internet, network threats are increasing, and the conventional security protection means cannot meet the requirements of enterprise security defense. The forms of logs, alarms, business data and the like related to network security are various, and how to effectively correlate to discover hidden threats is a hot spot problem focused by the industry.
However, in the related art, the method is only limited to real-time detection and monitoring of network nodes in the network service range, and threat discovery means are more dependent on single-point capability, cannot penetrate through upstream and downstream data capability and visual field, and cannot achieve the purpose of accurately and efficiently discovering threats due to insufficient visual field above threat discovery capability.
Therefore, how to accurately and efficiently discover the network threat is a problem to be solved at present.
Disclosure of Invention
Aiming at the problems existing in the prior art, the embodiment of the invention provides a system and a method for discovering the threat of full-flow security operation.
The invention provides a full-flow security operation threat discovery system, which comprises: the data access layer, the storage layer, the tool layer, the analysis layer and the service layer are sequentially arranged, and every two adjacent layers are connected with each other; wherein,
the data access layer is used for collecting multi-source information data from various data sources, preprocessing the multi-source information data and storing the multi-source information data into the storage layer;
the storage layer is used for classifying, storing and managing the multi-source information data based on at least one of category, level and sensitivity requirements, and constructing a structured data warehouse and a unstructured data lake;
The tool layer is used for carrying out deep mining and fusion on the multi-source information data in the data warehouse and the data lake based on a tool set to generate an information report and an information product;
the analysis layer is used for carrying out deep analysis on the external threat based on the information report and the information product, and generating threat assessment and early warning information of the external threat.
Optionally, the analysis layer includes:
the threat detection module is used for carrying out threat detection on the acquired network flow and log based on the information report and the information product, determining known and unknown external threats in the network flow and the log, and alarming for the external threats;
the task scheduling module is used for scheduling and managing the collection tasks of the multi-source information data;
the organization portrait analysis module is used for carrying out association analysis on the APT group archive, the APT threat attack weapon library and the white list, and constructing IP portraits and attack organization portraits aiming at each external threat;
the homology analysis module is used for extracting similarity analysis results of all external threats by carrying out similarity analysis on the IP portraits and attack organization portraits of all external threats; and determining homologous external threats based on the similarity analysis result, and generating threat assessment and early warning information of the homologous external threats.
Optionally, the system further comprises a service layer; the business layer and the analysis layer are connected with each other;
the business layer is used for continuously monitoring the threat assessment and early warning information, combining newly acquired multi-source information data and outputting situation awareness reports and tracking investigation reports.
Optionally, the service layer includes:
the threat situation module is used for continuously monitoring the threat assessment and early warning information and outputting the situation awareness report;
a rule operation module for storing rules, the rules comprising at least one of: data source acquisition rules, data processing rules, keyword filtering rules, APT detection rules and vulnerability detection rules;
the vulnerability discovery and emergency module is used for discovering various vulnerabilities in the process of collecting and analyzing the threat assessment and early warning information and outputting a vulnerability report and suggested repair measures;
and the tracking analysis module is used for continuously tracking and deeply analyzing the target attack objects, attack partners and events in the threat assessment and early warning information to generate the tracking investigation report.
Optionally, the data access layer includes:
the information acquisition module is used for acquiring multi-source information data from various data sources, wherein the multi-source information data comprises at least one of the following: full network asset information, open source information, business information, log audit system information, network flow data, mobile terminal data and host terminal data;
The data processing module is used for preprocessing the multi-source information data and storing the multi-source information data into the storage layer; the preprocessing of the multisource intelligence data comprises at least one of the following: classification, data normalization, deduplication, denoising.
Optionally, the tool layer includes:
and the tool set module is used for carrying out evidence collection, treatment and service recovery on the multi-source information data and generating the information report and the information product.
Optionally, the storage layer includes:
the information processing module is used for carrying out association analysis on the multi-source information data, establishing a private information base and updating various information data in the private information base; in the process of carrying out association analysis on the various information data, carrying out enrichment of labels and context data association;
a threat knowledge base module for storing data associated with the multi-source intelligence data and the external threat;
and the information storage module is used for storing the multi-source information data.
The invention also provides a method for discovering the threat of the whole-flow security operation, which comprises the following steps:
collecting multi-source information data from various data sources, and preprocessing the multi-source information data;
Classifying, storing and managing the multisource information data based on at least one of category, level and sensitivity requirements, and constructing a structured data warehouse and an unstructured data lake;
deep mining and fusion are carried out on the multi-source information data in the data warehouse and the data lake based on a tool set, so as to generate an information report and an information product;
and carrying out deep analysis on the external threat based on the information report and the information product to generate threat assessment and early warning information of the external threat.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the full-flow security operation threat discovery method according to any of the above when executing the program.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a full-flow security operation threat discovery method as described in any of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements a full-flow security operation threat discovery method as described in any of the above.
The invention provides a full-flow security operation threat discovery system and a full-flow security operation threat discovery method, wherein multi-source information data are collected from various data sources through a data access layer, preprocessed and stored in a storage layer; the storage layer carries out classified storage management on the multisource information data based on at least one of category, level and sensitivity requirements, and constructs a structured data warehouse and a unstructured data lake; then, deep mining and fusion are carried out on multi-source information data in a data warehouse and a data lake on the basis of a tool set in a tool layer, evidence collection, treatment and service recovery are carried out, and an information report and an information product are generated; deep analysis is carried out on external threats based on the information report and the information product in an analysis layer, and threat assessment and early warning information of the external threats are generated; by the system, the security situation can be accurately perceived aiming at various known and unknown threats in the network environment, and the effectiveness and the high efficiency of detection and threat discovery are further improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic diagram of a system architecture of a full-flow security operation threat discovery system provided by the present invention;
FIG. 2 is a second schematic diagram of a system architecture of the full-flow security operation threat discovery system provided by the present invention;
FIG. 3 is a schematic flow chart of data operation analysis provided by the present invention;
FIG. 4 is a third diagram of a system architecture of the full-flow security operation threat discovery system provided by the present invention;
FIG. 5 is a schematic logic flow diagram of a full-flow security operation threat discovery system provided by the invention;
FIG. 6 is a schematic diagram of the overall data flow of the full-flow security operation threat discovery system provided by the present invention;
FIG. 7 is a schematic flow chart of a full-flow security operation threat discovery method provided by the invention;
FIG. 8 is a schematic structural diagram of a full-flow security operation threat discovery apparatus provided by the invention;
fig. 9 is a schematic structural diagram of an electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
In order to facilitate a clearer understanding of the various embodiments of the present application, some relevant knowledge will be presented first.
With the rapid development of the internet, network threats are increasing, and the conventional security protection means cannot meet the requirements of enterprise security defense. In order to cope with the network threat, the existing security operation reflects the current security threat situation in time and accurately by using threat information processing, which is still a technical problem.
The forms of logs, alarms, service data and the like related to network security are various, and methods and means for discovering problems by approaching the service are needed for effectively associating the logs, alarms, service data and the like to discover hidden threats. The current threat discovery means are more dependent on single point capability, cannot penetrate upstream and downstream data capability and visual field, and cannot fully view the above threat discovery capability.
Specifically, in the prior art, there are the following problems:
current technology is limited to real-time detection and monitoring of network nodes within the network traffic to discover any possible attack cues and behaviors. However, most of data of service scenes cannot be better associated and analyzed, and the aim of efficiently discovering threats in the service is not achieved.
That is, based on the aspects of data type, data comprehensiveness and association analysis, only a single-point effect presentation supports the discovery of clues at present, and no good technical means is provided for threat tracking and advanced persistent attack (Advanced Persistent Threat, APT) organization malicious sample analysis. Among these, APT is a complex, persistent network attack in which an intruder establishes an undiscovered foothold in the network in order to steal sensitive data over a longer period of time.
Aiming at the problems, the invention provides a full-flow security operation threat discovery system which completes the processing of the full flow of 'prevention-discovery-detection-disposal-recovery-attribution' of network attack. And through multi-source security data acquisition of the data layer, the organization portrait analysis association engine and the vulnerability homology analysis engine of the analysis layer are utilized to perform data processing and association, so that vulnerability information analysis and malicious attack discovery are finally completed, potential threats in a network space are identified, and data support is provided for security operation decisions.
That is, the core of the full-flow security operation threat discovery system provided by the invention is that multi-source data (comprising business, open source information, mapping node information, access threat information, network flow information, log information and the like) are collected to a data lake and respectively enter different data types for storage. And meanwhile, the vulnerability attack homology attack and the tissue portrait analysis are completed at an analysis layer. And meanwhile, the tool set is utilized to carry out evidence collection, treatment, service recovery and the like, and finally, the capability of attack discovery and detection continuous tracking is achieved.
The full-flow security operation threat discovery system provided by the invention is specifically described below with reference to fig. 1. Fig. 1 is a schematic diagram of a system architecture of a full-flow security operation threat discovery system provided by the present invention, and referring to fig. 1, the full-flow security operation threat discovery system includes a data access layer 101, a storage layer 102, a tool layer 103, and an analysis layer 104, which are sequentially arranged, and two adjacent layers are connected with each other, wherein:
the data access layer 101 is configured to collect multi-source information data from various data sources, pre-process the multi-source information data, and store the multi-source information data in the storage layer 102.
In the embodiment of the present invention, the data access layer 101 collects and evidence-obtains the service data of all the key nodes and performs data summarization by accessing the log analysis module, thereby realizing collecting summarized information data from various data sources and preprocessing the multi-source information data.
In practical applications, the data access layer 101 is responsible for collecting structured and unstructured data from various data sources, performing preliminary formatting, cleaning and classification, and storing the data in the storage layer, thereby providing basic data support for subsequent processing and analysis. Note that the data access layer 101 focuses on the breadth, depth, and accuracy of data.
The storage layer 102 is configured to store and manage the multi-source intelligence data in a classified manner based on at least one of category, level and sensitivity requirements, and construct a structured data warehouse and an unstructured data lake.
In the embodiment of the present invention, the data access layer 101 aggregates various kinds of information data into object storage (Object Storage Service, OSS) and data lakes.
It should be noted that the storage layer 102 includes an operation rule base, a relational database, a non-relational database, a database cluster (e.g., ES cluster, S3 cluster), etc., performs data storage, data processing, and data association, and stores and associates different database types for different service data. Meanwhile, threat information after processing is stored, an information base is established, and labels, searching and management are provided.
In practical applications, the storage layer 102 performs classified storage management on the collected data according to the category, level and sensitivity requirements, and constructs a structured data warehouse and a unstructured data lake. The process focuses on efficient storage and management of mass data, and forms unified data processing and application.
Structured data warehouses are a database system for storing and analyzing large amounts of data, the data in structured databases typically being structured, following predefined models and metadata for querying and analysis.
Unstructured data lakes are a distributed storage system that is used to store and process large amounts of data. It is able to accept various forms of unstructured data, such as e-mail, documents, PDF.
The tool layer 103 is configured to deeply mine and fuse the multi-source information data in the data warehouse and the data lake based on a tool set, and generate an information report and an information product.
In the embodiment of the invention, technical tools such as multi-source data mining, machine learning, association analysis and the like are utilized to deeply mine and fuse multi-source information data, so that the internal connection among the data is found, and an information report and an information product are generated. The core of the process is to use technical means to realize value conversion.
In practical applications, the tools in the tool set include at least one of: the vulnerability emergency and detection, host vulnerability scanning, application vulnerability scanning, auditing and analysis and other tool sets complete data interaction and scheduling with the business layer and the analysis layer 104, and local analysis and evidence collection are realized.
The analysis layer 104 is configured to perform deep analysis on the external threat based on the information report and the information product, and generate threat assessment and early warning information of the external threat.
In the embodiment of the invention, the analysis layer 104 carries out deep analysis, association and judgment on the external threat by the processing result of the tool layer 103, and establishes threat assessment and early warning products. The process focuses on constructing the association relationship between data and mining threat indicators therein.
Specifically, in the analysis layer 104, the external threat is processed and detected through the use of a homologous analysis correlation engine and a organizational portrayal analysis engine. Analyzing the attack organization and the attack homology to obtain an attack organization portrait, and correlating the attack organization portrait with the network basic data and the information to obtain attack action analysis and TTP technical and tactical information.
And then carrying out gene extraction, dynamic and static scanning, malicious code homology analysis and attack vector library association analysis aiming at the file by combining an attack homology analysis method so as to achieve homology and map association of the sample and form attack homology analysis, reproduction and result and association. And finally, preparing threat assessment and early warning products of external threats.
According to the full-flow security operation threat discovery system, after the large-network mapping data and the timing task are adopted to access the open source and business information data, the data are summarized and processed, and the data fall into a data lake to be stored. The system architecture of the data access layer, the storage layer, the tool layer, the analysis layer and the service layer is adopted, and through acquisition, storage, processing, analysis, mining and knowledge management of mass data, the deep understanding of the mass data is realized, and the judgment of network space situation and threat is generated.
Optionally, fig. 2 is a second schematic diagram of a system architecture of the full-flow security operation threat discovery system provided by the present invention, and referring to fig. 2, the full-flow security operation threat discovery system includes a data access layer 101, a storage layer 102, a tool layer 103, an analysis layer 104, and a service layer 105, which are sequentially arranged, and are connected to each other between two adjacent layers, wherein:
the data access layer 101, the storage layer 102, the tool layer 103 and the analysis layer 104 are the same as those in fig. 1, and are not described here again to avoid repetition.
The business layer 105 is configured to continuously monitor the threat assessment and early warning information, and output a situation awareness report and a tracking investigation report in combination with newly collected multi-source information data.
The service layer 105 is used for performing service interaction and analysis support, and is responsible for processing the service operations such as the issuing of APT rules for processing attack information, the operation and updating of flow detection rules, the operation and updating of vulnerability detection rules, the updating of information, the issuing and the like.
In the embodiment of the invention, the business layer 105 continuously monitors threat assessment and early warning information, combines newly acquired multi-source information data, judges overall threat situation and change trend, discovers emerging threats and high-risk objects, and outputs situation awareness reports and tracking investigation reports. The process focuses on discovering vulnerabilities and threats in dynamic changes.
Furthermore, the full-flow security operation threat discovery system provided by the invention can also utilize deep learning, knowledge graph and homologous analysis technology to evaluate and judge the risk of the network space threat, and generate early warning information, report output, operation scheme recommendation and the like, thereby providing decision support for security operation. And meanwhile, analysis results, safety information, suggested API interfaces and other modes are provided and output to other operation management or product systems, so that information support is provided for decision making and response work.
In the process of generating the operation scheme recommendation, the flow of performing operation analysis on the data is shown in fig. 3. Fig. 3 is a schematic flow chart of data operation analysis provided by the present invention.
In fig. 3, thread discovery is performed based on data monitoring, customer feedback, darknet/big net monitoring, on-demand information data, and third party/open source data sources, and IP, domain name, vulnerability disclosure, data leakage, lux attack, etc. of each data are extracted, thereby generating event background.
Then, based on manual clue analysis and research, clue mining expansion, event continuous monitoring and clue positioning are carried out, and an attack technique is determined.
Then, carry out the investigation of manual work on-line, include: process analysis, network connection, account abnormity, system log, sensitive directory, evidence collection and retention, and related IOC sample flow is generated.
Then, performing cue tracing analysis, including: organization portrait analysis, vulnerability attack homology analysis, network basic information inquiry, threat information association inquiry and the like, thereby realizing attack tracing.
And finally, generating a threat event depth analysis report based on the event background, the attack manipulation, the related IOC sample flow and the attack tracing.
The full-flow security operation threat discovery system provided by the invention collects multi-source information data from various data sources through the data access layer, and pre-processes the multi-source information data and stores the multi-source information data in the storage layer; the storage layer carries out classified storage management on the multisource information data based on at least one of category, level and sensitivity requirements, and constructs a structured data warehouse and a unstructured data lake; then, deep mining and fusion are carried out on multi-source information data in a data warehouse and a data lake on the basis of a tool set in a tool layer, evidence collection, treatment and service recovery are carried out, and an information report and an information product are generated; deep analysis is carried out on external threats based on the information report and the information product in an analysis layer, and threat assessment and early warning information of the external threats are generated; finally, threat assessment and early warning information is continuously monitored in a business layer, newly collected multi-source information data is combined, situation awareness reports and tracking investigation reports are output, and the capability of attack discovery and detection continuous tracking is achieved; by the system, the security situation can be accurately perceived aiming at various known and unknown threats in the network environment, and the effectiveness and the high efficiency of detection and threat discovery are further improved.
FIG. 4 is a third diagram of a system architecture of the full-flow security operation threat discovery system provided by the present invention.
Optionally, the data access layer 101 includes:
the intelligence collection module 1011 is configured to collect multi-source intelligence data from various data sources, where the multi-source intelligence data includes at least one of: full network asset information, open source information, business information, log audit system information, network flow data, mobile end data, host end data.
In the embodiment of the invention, the information acquisition module 1011 accesses information such as real-time updated whole network asset information, open source information, business information, log audit system information, network flow data, mobile terminal data, host terminal data and the like through various paths.
The data processing module 1012 is configured to preprocess the multisource information data and store the multisource information data in the storage layer; the preprocessing of the multisource intelligence data comprises at least one of the following: classification, data normalization, deduplication, denoising.
In the embodiment of the invention, the data processing module 1012 classifies and normalizes the collected multi-source information data, and performs de-duplication and de-noising on the multi-source information data through task scheduling and stores the multi-source information data into various databases of different types to form a unified data lake.
Optionally, the storage layer 102 includes:
the information processing module 1021 is used for carrying out association analysis on the multi-source information data, establishing a private information base and updating various information data in the private information base; and in the process of carrying out association analysis on the various types of information data, enriching labels and associating context data.
In the embodiment of the present invention, the information processing module 1021 performs association analysis on multi-source information data, such as threat information, IP portrait information, internet base data, APT attack organization information, etc., distributed by cloud operation, records threat information generated by processing, establishes a private information library, evaluates timeliness of information stored in the private information library, and performs periodic update and research and judgment treatment. And meanwhile, in the process of carrying out association analysis on various types of information data, enriching labels and associating context data.
Threat knowledge base module 1022 is configured to store data associated with the multi-source intelligence data and the external threat.
In the embodiment of the present invention, the data associated with the multisource information data and the external threat includes whole network asset information, vulnerability asset information, information, multisource information, white list information, etc., and the data access and processing are completed through the information acquisition scheduling component, and the data application is completed to the storage layer 102.
And an intelligence storage module 1023 for storing the multi-source intelligence data.
Optionally, the tool layer 103 includes:
and the tool set module 1031 is used for performing evidence collection, treatment and service recovery on the multi-source information data and generating the information report and the information product.
In the embodiment of the present invention, the tool set module 1031 completes the tool query capabilities such as local evidence collection, threat discovery, research, treatment, cloud linkage query, and the like through various tool sets.
It should be noted that the tool set includes at least one of the following tools: vulnerability emergency and detection tool, host vulnerability scanning tool, application vulnerability scanning tool and audit and analysis tool.
Optionally, the analysis layer 104 includes:
the threat detection module 1041 is configured to perform threat detection on the obtained network traffic and log based on the information report and the information product, determine known and unknown external threats in the network traffic and the log, and alarm for the external threats.
The task scheduling module 1042 is used for scheduling and managing the collection task of the multi-source information data.
In the embodiment of the present invention, the task scheduling module 1042 is responsible for scheduling and managing various information gathering tasks, automatically executing the information gathering tasks according to a predetermined time period, and monitoring the task execution status and result to ensure the normal operation of important tasks.
While providing the functionality of manually triggering specific tasks, and integrating and mapping data through management scheduling.
The organization portrait analysis module 1043 is configured to perform association analysis on the advanced persistent attack APT partner archive, the APT threat attack weapon library and the white list, and construct an IP portrait and an attack organization portrait for each external threat.
In an embodiment of the present invention, the organization portrayal analysis module 1043 performs an omnibearing analysis and portrayal on a specific organization by performing a correlation analysis on the APT partner archive, the APT threat attack weapon repository and the white list, and by data collected from the multisource public information. The analysis content comprises basic information, personnel information, a relation network and the like, an IP portrait and an attack organization portrait are constructed, and the interaction mode and the characteristics of the organization and the interested relatives are revealed.
And adopting a multi-analysis engine to carry out omnibearing analysis mining and association on the specific target, and constructing an exhaustive portrait of the target.
The specific method for constructing the IP portrait and the attack organization portrait comprises the following steps: through the detection and analysis of each data meta information extracted from the data lake and by combining with the APT group archive, the white list and the threat attack weapon library, the association analysis is carried out, so that the IP portrait and the attack organization portrait are constructed. And (3) performing research and judgment and tag calibration from time, event, clue and network basic data association parts, and giving attack image results and used technical and tactical information for research and judgment and threat positioning.
The homology analysis module 1044 extracts a similarity analysis result of each external threat by performing similarity analysis on the IP portraits and attack organization portraits of each external threat; and determining homologous external threats based on the similarity analysis result, and generating threat assessment and early warning information of the homologous external threats.
In the embodiment of the invention, the correlation among various data is analyzed by carrying out static code similarity analysis and correlation analysis engine judgment on the IP portrait of the external threat and the attack organization portrait, and the attack homology analysis of different data after dynamic and static scanning is judged, so that basic support is provided for subsequent correlation analysis and data fusion.
The specific treatment method comprises the following steps:
1. the IP representation and the attack organization representation are input to a homology analysis module 1044.
2. Through filtering and delivering identification of IP portraits and attack organization portraits (hereinafter referred to as samples), file static scanning and dynamic analysis are carried out on the samples, and gene extraction, malicious code static characteristics and behavior similarity analysis are carried out on the samples aiming at sample meta-information.
3. Extracting a number similarity analysis result, and after correlation analysis by a correlation analysis engine, a knowledge graph analysis engine and the like, confirming threat assessment and early warning information (also called attack vector information) of homologous external threats, wherein the method comprises the following steps:
1) Attack targets: the sample data reflects the specific target that the attacker is aiming at, such as an operating system, software, system services or enterprise business, etc. This may help analyze the strategic intent and preferences of the attacker.
2) Attack technique: the technical and tactical information, the specific attack tools, attack methods and associated URL addresses used in the sample can analyze the technical resources and levels mastered by the attacker according to the information.
3) Attack path: an attacker represented by the sample data enters a path of the network or the system, such as through a web page vulnerability, USB intrusion, and the like. This helps determine the existence of a weak link in the enterprise network or system.
4) Attack source: information of an attacker, such as an IP address, a domain name server, a device fingerprint, etc., extracted from the sample. This helps determine the attribution and attributes of the attacker.
5) Attack time: the time point of sample generation or release is helpful for detecting the active period of attack activity, and predicting the high incidence period of next wave attack and the presentation of history information.
6) Attack surface influence: the impact of attacks that may be taken from the sample on traffic, data, such as implanted backdoors, stolen data, etc. This can reflect the destructive power and the degree of effort of the attacker.
Optionally, the service layer 105 includes:
the threat situation module 1051 is configured to continuously monitor the threat assessment and early warning information, and output the situation awareness report.
In the embodiment of the invention, the threat situation module 1051 analyzes and monitors threat assessment and early warning information to judge threat elements, threat events and trends in a service environment, assess the overall threat situation and predict the subsequent change trend; and further, a situation awareness report is output so as to improve threat awareness and coping capability of the user. The situation awareness report comprises threat reports, early warning information and the like.
A rule operation module 1052 for storing rules, the rules comprising at least one of: data source collection rules, data processing rules, keyword filtering rules, APT detection rules and vulnerability detection rules.
The vulnerability discovery and emergency module 1053 discovers various vulnerabilities in the process of acquiring and analyzing the threat assessment and early warning information, and outputs a vulnerability report and suggested repair measures.
In the embodiment of the invention, the vulnerability discovery and emergency module 1053 actively discovers various vulnerabilities and weaknesses existing in a system, a product or a service in the information acquisition and analysis process, and timely outputs a vulnerability report and suggested repair measures to ensure the information security of a user.
And the tracking analysis module 1054 is used for continuously tracking and deeply analyzing the target attack objects, attack partners and events in the threat assessment and early warning information to generate the tracking investigation report.
In an embodiment of the present invention, the tracking analysis module 1054 performs continuous tracking attention and in-depth analysis on specific target attack objects and attack partners and events (e.g., accents, specific events). And collecting relevant information, mining value information and intelligence, outputting a tracking investigation report, and revealing the activity rule of the object.
In practical application, the full-flow security operation threat discovery system provided by the invention can be used for carrying out security perception on an enterprise internal network, grasping the whole asset information of the intranet, and immediately utilizing a vulnerability emergency response to process an attack event after the threat attack event is discovered in the intranet, so as to treat and recover an abnormal system. In addition, in order to realize the omnibearing depth analysis of special malicious codes, the organization portrait and attack characteristic data are constructed by combining private threat information and a malicious file depth analysis module, so that the purpose of homologous analysis of attack events is achieved, and the establishment of a private information library and a rule library is enriched. By the method, the asset risk identification and the vulnerability threat discovery are cooperatively carried out, and aiming at various known and unknown threats in the network environment, the security situation is accurately perceived, so that the effectiveness of detection and threat discovery is further improved.
Fig. 5 is a logic flow diagram of the full-flow security operation threat discovery system provided by the invention, and see fig. 5 for a logic flow diagram:
and 1, data access.
Specifically, step 1 is responsible for collecting structured and unstructured data from various data sources, performing preliminary formatting, cleaning and classification, storing the data into a data storage platform, and providing basic data support for subsequent processing and analysis. This step focuses on the breadth, depth and accuracy of the data.
And 2, storing data.
Specifically, step 2 is responsible for carrying out classified storage management on the collected data according to the category, the level and the sensitivity requirement, and constructing a structured data warehouse and an unstructured data lake. The step focuses on efficient storage and management of mass data, and unified data processing and application are formed.
And 3, processing the tool set.
Specifically, step 3 uses technical tools such as multi-source data mining, machine learning, association analysis and the like to deeply mine and fuse stored data, discover internal relations among the data, and produce information reports and information products. The core of the step is to use technical means to realize value conversion.
And 4, threat analysis and association.
Specifically, step 4 carries out deep analysis, association and judgment on the tool set processing result and external threats, and establishes threat assessment and early warning products. This step focuses on constructing the association between data and mining threat indicators therein.
And 5, situation awareness and threat tracking.
Specifically, step 5 continuously monitors threat assessment and early warning information from step 4, combines new collected data, judges overall threat situation and change trend, discovers emerging threats and high-risk objects, and outputs situation awareness reports and tracking investigation reports. This step focuses on discovering vulnerabilities and threats in dynamic changes.
Referring to fig. 6, fig. 6 is an overall data flow schematic diagram of the full-flow security operation threat discovery system provided by the invention. Fig. 6 details the data flow between the layers in the full-flow security operation threat discovery system.
Fig. 7 is a flow chart of the full-flow security operation threat discovery method provided by the invention, and referring to fig. 7, the method includes steps 701-705, wherein:
step 701, multi-source information data are collected from various data sources, and preprocessing is carried out on the multi-source information data;
step 702, classifying, storing and managing the multi-source information data based on category, level and sensitivity requirements, and constructing a structured data warehouse and an unstructured data lake;
step 703, performing deep mining and fusion on the multi-source information data in the data warehouse and the data lake based on a tool set to generate an information report and an information product;
Step 704, carrying out deep analysis on external threats based on the information report and the information product, and generating threat assessment and early warning information of the external threats;
step 705, continuously monitoring the threat assessment and early warning information, and combining newly collected multi-source information data to output situation awareness reports and tracking investigation reports.
In the full-flow security operation threat discovery method provided by the invention, the security situation of the network environment can be comprehensively and timely perceived by converging each audit log, evidence obtaining log, alarm log and flow log of the intranet, the basis and tools are provided for security operators to accurately judge the network risk, and the targeted precaution strategies and measures are formulated, so that the security situation perception capability is improved;
the enterprise asset vulnerability information and the unknown threat are discovered through a new detection means, so that the prevention blind area of a safety operator in dealing with the unknown threat can be reduced to the greatest extent. The protection net in the safety system is more plump, continuous and leak-free, so that the prevention dead zone of unknown threats is reduced;
by carrying out context association analysis of each engine on heterogeneous security data and events, complex attack activities hidden under massive information can be found, internal association between security events is found, and the whole attack chain is captured. The skill and the level of the security operator in the aspects of threat hunting and attack tracing are obviously improved, and the threat association analysis capability is enhanced;
Meanwhile, the invention adopts a data lake mode to define a storage architecture, thereby realizing unified planning, scheduling, configuration and management of storage resources, simplifying operation, simplifying management and improving flexibility.
In conclusion, the method can discover, analyze and operate the threats in time, and greatly improves the efficiency of analysts and the accuracy of threat discovery.
The full-process security operation threat discovery device provided by the invention is described below, and the full-process security operation threat discovery device described below and the full-process security operation threat discovery method described above can be referred to correspondingly. Fig. 8 is a schematic structural diagram of a full-process security operation threat discovery apparatus provided by the invention, and as shown in fig. 8, the full-process security operation threat discovery apparatus 800 includes: the device comprises an acquisition module 801, a storage module 802, a first generation module 803 and a second generation module 804, wherein:
the acquisition module 801 is used for acquiring multi-source information data from various data sources and preprocessing the multi-source information data;
a storage module 802, configured to perform classified storage management on the multi-source intelligence data based on at least one of category, level and sensitivity requirements, and construct a structured data warehouse and an unstructured data lake;
A first generation module 803, configured to perform deep mining and fusion on the multi-source information data in the data warehouse and the data lake based on a tool set, and generate an information report and an information product;
the second generating module 804 is configured to perform deep analysis on the external threat based on the information report and the information product, and generate threat assessment and early warning information of the external threat.
The full-flow security operation threat discovery device provided by the invention collects multi-source information data from various data sources and preprocesses the multi-source information data; then classifying, storing and managing the multisource information data based on at least one of category, level and sensitivity requirements, and constructing a structured data warehouse and an unstructured data lake; then, carrying out deep mining and fusion on multi-source information data in the data warehouse and the data lake, and carrying out evidence collection, treatment and service recovery to generate an information report and an information product; then, based on the information report and the information product, carrying out deep analysis on the external threat to generate threat assessment and early warning information of the external threat; by the method, aiming at various known and unknown threats in the network environment, the security situation can be accurately perceived, and the effectiveness and the high efficiency of detection and threat discovery are further improved.
Optionally, the apparatus further comprises:
and the output module is used for continuously monitoring the threat assessment and early warning information, combining newly acquired multi-source information data and outputting situation awareness reports and tracking investigation reports.
Fig. 9 illustrates a physical schematic diagram of an electronic device, as shown in fig. 9, which may include: processor 910, communication interface (Communications Interface), memory 930, and communication bus 940, wherein processor 910, communication interface 920, and memory 930 communicate with each other via communication bus 940. Processor 910 may invoke logic instructions in memory 930 to perform a full-flow security operation threat discovery method comprising: collecting multi-source information data from various data sources, and preprocessing the multi-source information data; classifying, storing and managing the multisource information data based on at least one of category, level and sensitivity requirements, and constructing a structured data warehouse and an unstructured data lake; deep mining and fusion are carried out on the multi-source information data in the data warehouse and the data lake based on a tool set, so as to generate an information report and an information product; and carrying out deep analysis on the external threat based on the information report and the information product to generate threat assessment and early warning information of the external threat.
Further, the logic instructions in the memory 930 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product, where the computer program product includes a computer program, where the computer program can be stored on a non-transitory computer readable storage medium, and when the computer program is executed by a processor, the computer can execute a full-flow security operation threat discovery method provided by the above methods, and the method includes: collecting multi-source information data from various data sources, and preprocessing the multi-source information data; classifying, storing and managing the multisource information data based on at least one of category, level and sensitivity requirements, and constructing a structured data warehouse and an unstructured data lake; deep mining and fusion are carried out on the multi-source information data in the data warehouse and the data lake based on a tool set, so as to generate an information report and an information product; and carrying out deep analysis on the external threat based on the information report and the information product to generate threat assessment and early warning information of the external threat.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the full-flow security operation threat discovery method provided by the methods above, the method comprising: collecting multi-source information data from various data sources, and preprocessing the multi-source information data; classifying, storing and managing the multisource information data based on at least one of category, level and sensitivity requirements, and constructing a structured data warehouse and an unstructured data lake; deep mining and fusion are carried out on the multi-source information data in the data warehouse and the data lake based on a tool set, so as to generate an information report and an information product; and carrying out deep analysis on the external threat based on the information report and the information product to generate threat assessment and early warning information of the external threat.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (12)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311380230.0A CN117640142A (en) | 2023-10-23 | 2023-10-23 | Full-process security operation threat discovery system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311380230.0A CN117640142A (en) | 2023-10-23 | 2023-10-23 | Full-process security operation threat discovery system and method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117640142A true CN117640142A (en) | 2024-03-01 |
Family
ID=90018855
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311380230.0A Pending CN117640142A (en) | 2023-10-23 | 2023-10-23 | Full-process security operation threat discovery system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117640142A (en) |
-
2023
- 2023-10-23 CN CN202311380230.0A patent/CN117640142A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Log2vec: A heterogeneous graph embedding based approach for detecting cyber threats within enterprise | |
| US20230011004A1 (en) | Cyber security sandbox environment | |
| CN116662989B (en) | Security data analysis method and system | |
| Gupta et al. | Layered approach using conditional random fields for intrusion detection | |
| US8375452B2 (en) | Methods for user profiling for detecting insider threats based on internet search patterns and forensics of search keywords | |
| CN110602042B (en) | APT attack behavior analysis and detection method and device based on cascade attack chain model | |
| CN113486351A (en) | Civil aviation air traffic control network safety detection early warning platform | |
| Deepa et al. | A comprehensive survey on approaches to intrusion detection system | |
| CN109347808B (en) | Safety analysis method based on user group behavior activity | |
| CN118536093B (en) | Data security tracing method, system and device based on artificial intelligence | |
| CN113642023A (en) | Data security detection model training method, data security detection device and equipment | |
| CN113923037B (en) | Anomaly detection optimization device, method and system based on trusted computing | |
| Datta et al. | Real-time threat detection in ueba using unsupervised learning algorithms | |
| Jiang et al. | Warder: Online insider threat detection system using multi-feature modeling and graph-based correlation | |
| CN117454376A (en) | Industrial Internet data security detection response and tracing method and device | |
| CN117375997A (en) | Malicious traffic attack security knowledge plane construction method based on honey points | |
| CN119205351A (en) | A method and device for tracking and tracing asset risks | |
| CN114500122B (en) | Specific network behavior analysis method and system based on multi-source data fusion | |
| KR102592624B1 (en) | Threat hunting system and method for against social issue-based advanced persistent threat using artificial intelligence | |
| CN119276604A (en) | A monitoring method for cloud security system | |
| Wen et al. | Detecting and predicting APT based on the study of cyber kill chain with hierarchical knowledge reasoning | |
| CN117220961B (en) | An intrusion detection method, device and storage medium based on association rule graph | |
| CN113572781A (en) | Method for collecting network security threat information | |
| Pangsuban et al. | A real-time risk assessment for information system with cicids2017 dataset using machine learning | |
| CN117640142A (en) | Full-process security operation threat discovery system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |