CN111770106A - Method, device, system, electronic device and storage medium for data threat analysis - Google Patents
Method, device, system, electronic device and storage medium for data threat analysis Download PDFInfo
- Publication number
- CN111770106A CN111770106A CN202010644148.4A CN202010644148A CN111770106A CN 111770106 A CN111770106 A CN 111770106A CN 202010644148 A CN202010644148 A CN 202010644148A CN 111770106 A CN111770106 A CN 111770106A
- Authority
- CN
- China
- Prior art keywords
- data
- real
- data source
- time
- threat
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1433—Vulnerability analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0635—Risk analysis of enterprise or organisation activities
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
Abstract
The application relates to a method, a device, a system, an electronic device and a storage medium for data threat analysis, wherein the method for data threat analysis comprises the following steps: dividing the data source according to the data evaluation index, and acquiring a real-time data source in the data source according to the division result; based on a real-time calculation engine, acquiring alarm data of the real-time data source; obtaining a threat analysis result of the data source according to the alarm data and the threat assessment index; by the method and the device, the problem of high cost of data threat analysis is solved, and comprehensive optimization of the use efficiency of resources is realized.
Description
Technical Field
The present application relates to the field of data security technologies, and in particular, to a method, an apparatus, a system, an electronic apparatus, and a storage medium for data threat analysis.
Background
On a big data service platform, a lot of service data can be collected and stored; the massive service data may contain some threat data with potential safety hazards; in the related technology, the whole data is generally collected, processed and analyzed uniformly, but part of the data has poor quality, low effectiveness and low use degree, and occupies a large amount of calculation and storage resources; or the big data threat analysis is carried out by a centralized stream computing or batch computing (incremental) technology, the quantity and the performance of the servers are completely depended on, and the quantity of data sources is huge in application scenes such as the Internet of things, so that the requirements on the performance and the quantity of the servers are higher; therefore, data threat analysis in the related art fails to optimize server performance and storage resources, resulting in higher costs.
At present, no effective solution is provided for the problem of high cost of data threat analysis in the related technology.
Disclosure of Invention
The embodiment of the application provides a method, a device, a system, an electronic device and a storage medium for data threat analysis, so as to at least solve the problem of high cost of data threat analysis in the related technology.
In a first aspect, an embodiment of the present application provides a method for data threat analysis, where the method includes:
dividing data sources according to the data evaluation indexes, and acquiring real-time data sources in the data sources according to the dividing results;
acquiring alarm data of the real-time data source based on a real-time calculation engine;
and acquiring a threat analysis result of the data source according to the alarm data and the threat assessment index.
In a possible embodiment, the dividing the input data sources according to the data evaluation index and acquiring the real-time data source in the data sources according to the dividing result includes:
dividing the data source into a real-time data source and an off-line data source according to the data evaluation index; wherein the data evaluation indicator comprises at least one of: data threat, data timeliness requirements, and data relevancy requirements.
In a possible embodiment, after obtaining the threat analysis result of the data source according to the alarm data and the threat assessment index, the method further includes:
acquiring the priority level of a threat event in the threat analysis result; and under the condition that the priority level of the threat event is high, acquiring the offline data source matched with the threat event.
In a possible embodiment, after the dividing the data source into the real-time data source and the offline data source, and before the obtaining, based on the real-time computing engine, the alarm data of the real-time data source, the method further includes:
based on a data acquisition algorithm, sending the real-time data source to a message queue;
and based on the data warehouse technology, the offline data source is sent to a local storage device or a backup data server.
In a possible embodiment, after the obtaining of the real-time data source in the data sources according to the result of the division and before the obtaining of the alarm data of the real-time data source based on the real-time computing engine, the method further includes:
performing single data source association on the data source based on a data association algorithm, and acquiring a threat index or an alarm log of the data source; wherein the data association algorithm comprises at least one of: the method comprises the steps of loss index (IoC for short) real-time collision matching, Attack index (IoA for short) real-time collision matching, statistical threshold triggering and time window correlation triggering.
In a possible embodiment, after obtaining the threat analysis result of the data source according to the alarm data and the threat assessment index, the method further includes:
acquiring IoC and IoAs of threat events in the threat analysis results, and synchronizing the IoC and the IoAs to the data source; wherein the IoC is used for the IoC real-time collision matching and the IoA is used for the IoA real-time collision matching.
In a second aspect, an embodiment of the present application provides an apparatus for data threat analysis, where the apparatus includes: the system comprises a dividing module, a real-time computing module and a threat analyzing module;
the dividing module is used for dividing the data sources according to the data evaluation indexes and acquiring real-time data sources in the data sources according to the dividing result;
the real-time computing module is used for acquiring alarm data of the real-time data source based on a real-time computing engine;
and the threat analysis module is used for acquiring a threat analysis result of the data source according to the alarm data and the threat assessment index.
In a third aspect, an embodiment of the present application provides a system for data threat analysis, where the system includes: a terminal and a server;
the server divides the data source received by the terminal according to the data evaluation index and acquires a real-time data source in the data source according to the division result;
the server acquires alarm data of the real-time data source based on a real-time calculation engine;
and the server acquires a threat analysis result of the data source according to the alarm data and the threat assessment index.
In a fourth aspect, embodiments of the present application provide an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor, when executing the computer program, implements the method of data threat analysis as described in the first aspect above.
In a fifth aspect, the present application provides a storage medium, on which a computer program is stored, which when executed by a processor, implements the method for data threat analysis as described in the first aspect above.
Compared with the related art, the method, the device, the system, the electronic device and the storage medium for analyzing the data threat provided by the embodiment of the application divide the data source according to the data evaluation index, and acquire the real-time data source in the data source according to the dividing result; based on a real-time calculation engine, acquiring alarm data of the real-time data source; and acquiring a threat analysis result of the data source according to the alarm data and the threat assessment index, solving the problem of high cost of data threat analysis, and realizing comprehensive optimization of the use efficiency of resources.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a schematic diagram of an application scenario of a data threat analysis method according to an embodiment of the present invention;
FIG. 2 is a flow diagram of a data threat analysis method according to an embodiment of the application;
FIG. 3 is a schematic diagram of a data threat analysis architecture according to the related art;
FIG. 4 is a flow diagram of another method of data threat analysis according to an embodiment of the present application;
FIG. 5 is a flow diagram of yet another method for data threat analysis according to an embodiment of the present application;
FIG. 6 is a flow chart of yet another data threat analysis method according to an embodiment of the present application;
FIG. 7 is a block diagram of a data threat analysis apparatus according to an embodiment of the present application;
fig. 8 is a block diagram of the inside of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference herein to "a plurality" means greater than or equal to two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.
In the present embodiment, an application scenario of a method for data threat analysis is provided, and fig. 1 is a schematic diagram of an application scenario of a method for data threat analysis according to an embodiment of the present invention, as shown in fig. 1, in the application environment, a terminal 12 communicates with a server 14 through a network; the server 14 acquires the data source of the terminal 12, and acquires a real-time data source in the data source based on the data evaluation index; the server 14 obtains the alarm data of the real-time data source based on the real-time computing engine, and obtains the threat analysis result of the data source according to the alarm data and the threat assessment index; the terminal 12 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices, and the server 14 may be implemented by an independent server or a server cluster composed of a plurality of servers.
In this embodiment, a method for data threat analysis is provided, and fig. 2 is a flowchart of a method for data threat analysis according to an embodiment of the present application, and as shown in fig. 2, the flowchart includes the following steps:
step S202, dividing the data source according to the data evaluation index, and then acquiring a real-time data source in the data source according to the dividing result; the method comprises the following steps that data quality and effectiveness evaluation is conducted on each type of data source accessed by a big data system; the data source includes: network mirror flow, operating system logs, application logs, security equipment alarm logs, middleware logs, database logs and the like; classifying the input data sources according to the data types;
step S204, based on the real-time calculation engine, acquiring alarm data of the real-time data source; loading a predefined analysis model by using a big data real-time computing component or engine, such as a flash, an Apache storm and the like, reading real-time data flow in a Kafka cluster to generate alarm or statistical index data through computing, and uniformly storing computing result data to a big data platform, such as an ElasticSearch cluster; meanwhile, original data in the Kafka cluster are synchronously stored to the big data platform;
step S206, obtaining a threat analysis result of the data source according to the alarm data and the threat assessment index; the method comprises the following steps of analyzing alarm data obtained by calculating big data in real time and carrying out threat event risk assessment, wherein the referential threat assessment indexes comprise: attack type, attack frequency, attack results, IoC number, attack chain coverage, asset vulnerability and asset importance, etc. for threat event assessment based on the perspective of the attacker or the risky asset; and the weight value corresponding to each threat assessment index can be adjusted according to the actual situation.
Fig. 3 is a schematic diagram of a data threat analysis architecture according to the related art, and as shown in fig. 3, data support is finally provided for applications such as threat warning, source tracing investigation, situation awareness and the like by collecting, calculating, storing and applying log and flow data generated by a data source and combining a real-time and offline threat analysis model; the existing architecture supports a big data platform and a big data assembly, and the existing architecture commonly comprises: hadoop, HBase, Hive, Kafka, ElasticSearch, Flink, Spark, Storm, etc.; however, considering the rapid increase in the number of data sources, the leap in data quality and data volume, resulting in satisfying large data analysis processing by way of horizontal capacity expansion, is not the most cost-effective means;
compared with the related art, in the embodiment of the application, through the steps S202 to S206, a hybrid computing and storage architecture capable of processing massive data sources is set through data type division, and a threat analysis method and a data processing flow are more effectively obtained according to a new architecture design, so that data threats are analyzed, and the effects of optimizing the use condition of computing and storage resources of the server 14 and reducing the load of a big data cluster are achieved, so that the problem of high cost of a data threat method is solved, and comprehensive optimization of the use efficiency of resources is realized.
In one possible embodiment, the data source is divided into the real-time data source and an offline data source according to the data evaluation index; the data evaluation index includes at least one of: data threat level, data timeliness requirements and data relevance requirements;
wherein, the threat degree of the data refers to whether the data can directly or indirectly indicate an attack event or a system vulnerability risk; for example: the terminal 12 divides the data source into real-time data sources if the alarm log and the flow alarm log are data with large threat degree and need to be collected in real time; the audit data, the flow protocol audit data and the flow session data of the terminal 12 are data with low threat degree, can be stored locally, and are collected in a delayed manner, and then the data source is divided into an offline data source;
the data timeliness requirement refers to whether the type of data needs to be acquired, analyzed and threatened to alarm or has other real-time application requirements; for example: service access request data in the flow data need to calculate statistical indexes and analyze abnormal access behaviors in real time, the timeliness requirement is high, and a real-time acquisition mode needs to be adopted; file information data (original files are restored in flow) needs to be combined with offline analysis, such as a dynamic sandbox or antivirus software, so that the file information data can be stored locally and collected in a delayed manner;
the data correlation requirement refers to the requirement whether the type of data has real-time data correlation analysis with other data; performing correlation analysis on data sources needing to be crossed, wherein data of related data sources need to be acquired in real time; for example: the process network behavior data of the Windows terminal is combined with a malicious Domain Name alarm triggered by Domain Name System (DNS) query data in network traffic, so that malicious information infecting a host can be associated, and the two types of data need to be associated in real time.
Through the embodiment, all data sources are divided into real-time acquired data and offline acquired data based on a plurality of data evaluation indexes such as data threat degree, data timeliness requirements and data relevance requirements, so that the accuracy of data classification is improved, and the problem of resource allocation of different value data is solved.
In one possible embodiment, a method of data threat analysis is provided, and fig. 4 is a flowchart of another method of data threat analysis according to an embodiment of the present application, as shown in fig. 4, the flowchart includes the following steps:
step S402, obtaining the priority level of the threat event in the threat analysis result; the threat events are sorted according to the severity and the priority, and the threat events with high severity are arranged in the front, and the priority level corresponding to the threat events is set as the high level;
under the condition that the priority level of the threat event is high priority, acquiring the offline data source matched with the threat event; wherein, the offline collected data stored on the data source or the backup data server 14 is pulled for the threat event with high priority; threat data pulling can be completed by filtering data fields of related assets and attackers; for example: an attacker with an IP address of IP1 successfully launches a vulnerability attack on the IP2 server, and the remote code execution is successful, so that the threat event judgment priority is high, and other data related to IP1 and IP2, which are stored in a data source local or backup data server, need to be pulled for restoring the complete access behavior of IP1 to IP 2.
Through the step S402, the threat events in the threat analysis result are sorted according to the severity and the priority, and the related data stored offline is pulled for the high-priority threat events to be further analyzed, so that the thread tracing and the display of the threat events are performed, and the accuracy of the data threat analysis is improved.
In a possible embodiment, after the data source is divided into the real-time data source and the offline data source, before the real-time computing engine is used to obtain the alarm data of the real-time data source, the method for analyzing the data threat further includes the following steps:
based on a data acquisition algorithm, sending the real-time data source to a message queue; the real-time acquisition data obtained after single data source data association calculation and data classification are carried out on each data source, and a related data acquisition method is used, and comprises the following steps: self-research or open-source collector agents such as a syslog protocol, an Nxlog and the like, database table reading, FTP file transmission or other data acquisition methods, and sending the real-time data source to a message queue cache of a big data platform for real-time calculation and analysis; the message queue can be set as a Kafka cluster and the like;
based on the data warehouse technology, the offline data source is sent to a local storage device or a backup data server; wherein, for the offline collected data, a local program is adopted to perform Extract-Transform-Load (ETL for short) processing, and the ETL process comprises the following steps: data analysis, field extraction and addition, field value standardization, labeling according to built-in rules and the like; and after the processing is finished, the offline data source is sent to a local storage device for local temporary storage or sent to a backup data server.
Through the embodiment, the real-time data is sent to the message queue of the big data platform, and the local data of the offline data is stored or sent to the backup data server for storage, so that the data is separated and collected, the positions of the resources are separately calculated and stored according to actual requirements, and the use conditions of the calculation and storage resources of the server 14 are further optimized.
In one possible embodiment, a method for data threat analysis is provided, and fig. 5 is a flowchart of another data threat analysis method according to an embodiment of the present application, as shown in fig. 5, the flowchart includes the following steps:
step S502, performing single data source association on the data source based on a data association algorithm, and acquiring an alarm log of the data source; the data association algorithm comprises: IoC real-time collision matching, IoA real-time collision matching, same data type statistical threshold triggering, different data type time window correlation triggering and the like;
the correlation mode of IoC/IoA collision matching is as follows: the DNS query log is matched with a Command Control (C & C) domain name list in real time, and a new alarm log is output when a malicious domain name is hit, namely C & C domain name query;
the correlation mode of the statistical threshold trigger is as follows: counting the number of remote login failure logs in a Windows event within a period of time; for example, the remote login failure log is event ID 4625; 10, if the number of the logs exceeds a preset threshold value within 1 minute, outputting a new alarm log, namely, violently cracking a Remote Desktop Protocol (RDP);
the time window association triggering mode is as follows: within a time window (e.g., 1 minute) 2 associated logs are generated in succession, for example: and if the log 1 is an RDP brute force cracking alarm log, and the log 2 is a windows remote login success event, outputting a new alarm log, namely, the RDP brute force cracking is successful.
In one possible embodiment, after obtaining the threat analysis result, the method for data threat analysis further includes the following steps: acquiring IoC and IoA of the threat events in the threat analysis result, and synchronizing IoC and IoA to the data source; wherein the IoC is used for the IoC real-time collision matching, the IoA is used for the IoA real-time collision matching; thus, IoC/IoA is synchronized to the data source for association analysis on a single data source, and server 14 calculation is further optimized.
The following describes an embodiment of the present invention in detail with reference to an actual application scenario, and fig. 6 is a flowchart of another data threat analysis method according to the embodiment of the present application, where the data threat analysis method may be applied to collect Windows terminal data and network mirror image traffic data, and the specific implementation steps are as shown in fig. 6:
step S602, performing data association on a single data source; performing data correlation calculation on the same or different types of data of each data source accessed by the big data system to generate threat indexes or alarm logs of the data sources;
step S604, data classification; performing data quality and effectiveness evaluation on each type of data source, and dividing the type of data source into a real-time data source and an offline data source according to indexes such as data threat degree, data timeliness requirements and data relevance requirements;
step S606, data are separated and collected; the method comprises the following steps that a Json format log is sent on a Windows terminal by using an Nxlog program, a network flow collector also sends the log in the json format, and a real-time data source is sent to a Kafka cluster of a big data platform; ETL processing is carried out on the offline acquired data by adopting a local program, and local temporary storage is carried out after the processing is finished, namely a Windows terminal stores a local file, a flow collector stores a local database, or the offline acquired data is sent to a backup data server;
step S608, calculating and storing big data in real time; loading a predefined analysis model by using a big data real-time computing component or engine, such as flash, storm and the like, reading real-time data flow in the kafka cluster to generate alarm or statistical index data through computing, and uniformly storing computing result data to a big data platform such as an ElasticSearch cluster and the like; meanwhile, the original data in Kafka is synchronously stored in an ElasticSearch;
step S610, threat event evaluation; alarm data obtained by analyzing big data and calculating in real time is evaluated based on the views of attackers or risk assets and the like, and the evaluation indexes which can be referred to comprise: attack type, attack frequency, attack result, IoC number, attack chain coverage, asset vulnerability, asset importance, etc.; the weight can be adjusted according to the actual situation; obtaining the scores of the threat events and sequencing the threat events according to the severity and the priority;
step S612, delay pulling of off-line collected data; and pulling the offline collected data stored on the data source or the data backup server for the threat event with high priority for tracing and displaying the clues of the threat event. Threat data pulling can be completed by filtering data fields of related assets and attackers; for example: the attacker (IP address: IP1) successfully launches the remote code vulnerability execution attack to the server 14(IP address: IP2), the threat event judgment priority is high, and other data which are stored on a data source local or a data backup server and are related to IP1 and IP2 need to be pulled for restoring the complete access behavior of the IP1 to the IP 2;
steps S614, IoC are synchronous with IoA; the confirmed threat event extraction IoC and IoA are synchronized to all data sources for real-time collision matching of subsequent data.
It should be noted that, a big data analysis system architecture in the related art is constructed based on a centralized server or a cloud server cluster, and under the condition that a mass data source is involved, the collection, transmission, calculation and storage of the full data put a great stress on the server cluster, and meanwhile, due to different quantity and quality differences of the original data, the requirements on instantaneity, availability and effectiveness are inconsistent, and a new processing method and a system architecture are needed to comprehensively optimize the use efficiency of resources; in the embodiment of the present application, through the steps S602 to S614, the centralized big data computing and storing manner is improved to a hybrid processing architecture that utilizes the data source local and the backup data server resource, a big data threat analysis architecture that is hybrid computing and storing is constructed, an existing common big data analysis system architecture is optimized, and an improved data threat analysis method is provided based on a new architecture.
It should be understood that, although the steps in the flowcharts of fig. 2, 4 to 6 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2, 4 through 6 may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least some of the sub-steps or stages of other steps.
In this embodiment, a data threat analysis apparatus is provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, which have already been described and are not described again. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 7 is a block diagram of a data threat analysis apparatus according to an embodiment of the present application, and as shown in fig. 7, the apparatus includes: a partitioning module 72, a real-time computation module 74, and a threat analysis module 76;
the dividing module 72 is configured to divide the data source according to the data evaluation index, and obtain a real-time data source in the data source according to a dividing result; the real-time computing module 74 is configured to obtain alarm data of the real-time data source based on a real-time computing engine; the threat analysis module 76 is configured to obtain a threat analysis result of the data source according to the alarm data and the threat assessment indicator.
Through the embodiment, the data types are divided by the dividing module 72, a hybrid computing and storing architecture capable of processing the mass data sources is arranged, the threat analysis method and the data processing flow are more effectively obtained according to the new architecture design, the data threats are further analyzed, the effects of optimizing the use condition of computing and storing resources of the server 14 and reducing the load of a big data cluster are achieved, the problem of high cost of the data threat method is solved, and the comprehensive optimization of the use efficiency of the resources is realized.
In a possible embodiment, the dividing module 72 is further configured to divide the data source into the real-time data source and the offline data source according to the data evaluation index; wherein the data evaluation index comprises at least one of: data threat, data timeliness requirements, and data relevancy requirements.
In one possible embodiment, the apparatus for data threat analysis further comprises an offline module; the threat analysis module 76 is further configured to obtain a priority level of a threat event in the threat analysis result; the offline module obtains the offline data source matched with the threat event when the priority level of the threat event is a high priority level.
In one possible embodiment, the data threat analysis apparatus further comprises an acquisition module; the acquisition module is used for sending the real-time data source to a message queue based on a data acquisition algorithm; the acquisition module sends the offline data source to a local storage device or a backup data server based on data warehouse technology.
In one possible embodiment, the apparatus for data threat analysis further comprises an association module; the association module is used for performing single data source association on the data source based on a data association algorithm and acquiring an alarm log of the data source; wherein the data association algorithm comprises at least one of: IoC real-time collision match, IoA real-time collision match, statistical threshold trigger, and time window association trigger.
In one possible embodiment, the apparatus for data threat analysis further comprises a synchronization module; the synchronization module is used for acquiring IoC and IoA of the threat events in the threat analysis result and synchronizing the IoC and the IoA to the data source; wherein, the IoC is used for the IoC real-time collision matching, and the IoA is used for the IoA real-time collision matching.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
In the present embodiment, a system for data threat analysis is provided, which includes a terminal 12 and a server 14; the server 14 divides the data source received by the terminal 12 according to the data evaluation index, and obtains a real-time data source in the data source according to the division result; the server 14 obtains the alarm data of the real-time data source based on the real-time computing engine; the server 14 obtains a threat analysis result of the data source according to the alarm data and the threat assessment index.
Through the embodiment, the server 14 sets the hybrid computing and storing architecture capable of processing the mass data sources by dividing the data types, and more effectively obtains the threat analysis method and the data processing flow according to the new architecture design, so as to analyze the data threats, thereby achieving the effects of optimizing the use condition of the computing and storing resources of the server 14 and reducing the load of a big data cluster, solving the problem of high cost of the data threat method, and realizing the comprehensive optimization of the use efficiency of the resources.
In a possible embodiment, the server 14 is further configured to divide the data source into the real-time data source and the offline data source according to the data evaluation index; wherein the data evaluation index comprises at least one of: data threat, data timeliness requirements, and data relevancy requirements.
In one possible embodiment, the server 14 is further configured to obtain a priority level of the threat event in the threat analysis result; and under the condition that the priority level of the threat event is high priority, acquiring the offline data source matched with the threat event.
In one possible embodiment, the server 14 is further configured to send the real-time data source to a message queue based on a data collection algorithm; based on the data warehouse technology, the off-line data source is sent to the local storage device or the backup data server
In a possible embodiment, the server 14 is further configured to perform single data source association on the data source based on a data association algorithm, and obtain an alarm log of the data source; wherein the data association algorithm comprises at least one of: IoC real-time collision match, IoA real-time collision match, statistical threshold trigger, and time window association trigger.
In one possible embodiment, the server 14 is further configured to obtain IoC and IoA of threat events in the threat analysis results, and synchronize the IoC and the IoA to the data source; wherein, the IoC is used for the IoC real-time collision matching, and the IoA is used for the IoA real-time collision matching.
In some embodiments, a computer device is provided, and the computer device may be a server, and fig. 8 is a structural diagram of the inside of the computer device according to the embodiment of the present invention, as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used to store data sources. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of data threat analysis.
Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
The present embodiment also provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
step S1, dividing the data source according to the data evaluation index, and obtaining the real-time data source in the data source according to the dividing result;
step S2, based on the real-time calculation engine, obtaining the alarm data of the real-time data source;
and step S3, obtaining the threat analysis result of the data source according to the alarm data and the threat assessment index.
It should be noted that, for specific examples in this embodiment, reference may be made to examples described in the foregoing embodiments and optional implementations, and details of this embodiment are not described herein again.
In addition, in combination with the data threat analysis method in the foregoing embodiment, the embodiment of the present application may provide a storage medium to implement. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the data threat analysis methods of the above embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It should be understood by those skilled in the art that various features of the above-described embodiments can be combined in any combination, and for the sake of brevity, all possible combinations of features in the above-described embodiments are not described in detail, but rather, all combinations of features which are not inconsistent with each other should be construed as being within the scope of the present disclosure.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of data threat analysis, the method comprising:
dividing data sources according to the data evaluation indexes, and acquiring real-time data sources in the data sources according to the dividing results;
acquiring alarm data of the real-time data source based on a real-time calculation engine;
and acquiring a threat analysis result of the data source according to the alarm data and the threat assessment index.
2. The method of claim 1, wherein the dividing the input data sources according to the data evaluation index and acquiring the real-time data sources from the data sources according to the dividing result comprises:
dividing the data source into a real-time data source and an off-line data source according to the data evaluation index; wherein the data evaluation indicator comprises at least one of: data threat, data timeliness requirements, and data relevancy requirements.
3. The method of claim 2, wherein after obtaining the threat analysis results of the data source based on the alarm data and the threat assessment indicators, the method further comprises:
acquiring the priority level of a threat event in the threat analysis result; and under the condition that the priority level of the threat event is high, acquiring the offline data source matched with the threat event.
4. The method of claim 2, wherein after the dividing the data source into the real-time data source and the offline data source, and before the obtaining the alarm data of the real-time data source based on the real-time computing engine, the method further comprises:
based on a data acquisition algorithm, sending the real-time data source to a message queue;
and based on the data warehouse technology, the offline data source is sent to a local storage device or a backup data server.
5. The method of claim 1, wherein after the obtaining of the real-time data source from the data sources according to the partitioning result and before the obtaining of the alarm data of the real-time data source based on the real-time computing engine, the method further comprises:
performing single data source association on the data source based on a data association algorithm, and acquiring an alarm log of the data source; wherein the data association algorithm comprises at least one of: the method comprises the steps of real-time collision matching of the collapse index, real-time collision matching of the attack index, triggering of a statistical threshold value and correlation triggering of a time window.
6. The method of claim 5, wherein after obtaining the threat analysis results of the data source based on the alarm data and the threat assessment indicators, the method further comprises:
acquiring a missing index and an attack index of a threat event in the threat analysis result, and synchronizing the missing index and the attack index to the data source; the collapse index is used for real-time collision matching of the collapse index, and the attack index is used for real-time collision matching of the attack index.
7. An apparatus for data threat analysis, the apparatus comprising: the system comprises a dividing module, a real-time computing module and a threat analyzing module;
the dividing module is used for dividing the data sources according to the data evaluation indexes and acquiring real-time data sources in the data sources according to the dividing result;
the real-time computing module is used for acquiring alarm data of the real-time data source based on a real-time computing engine;
and the threat analysis module is used for acquiring a threat analysis result of the data source according to the alarm data and the threat assessment index.
8. A system for data threat analysis, the system comprising: a terminal and a server;
the server divides the data source received by the terminal according to the data evaluation index and acquires a real-time data source in the data source according to the division result;
the server acquires alarm data of the real-time data source based on a real-time calculation engine;
and the server acquires a threat analysis result of the data source according to the alarm data and the threat assessment index.
9. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is configured to execute the computer program to perform the method of data threat analysis of any one of claims 1 to 6.
10. A storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the method of data threat analysis of any one of claims 1 to 6 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010644148.4A CN111770106A (en) | 2020-07-07 | 2020-07-07 | Method, device, system, electronic device and storage medium for data threat analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010644148.4A CN111770106A (en) | 2020-07-07 | 2020-07-07 | Method, device, system, electronic device and storage medium for data threat analysis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111770106A true CN111770106A (en) | 2020-10-13 |
Family
ID=72723928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010644148.4A Pending CN111770106A (en) | 2020-07-07 | 2020-07-07 | Method, device, system, electronic device and storage medium for data threat analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111770106A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685394A (en) * | 2020-12-25 | 2021-04-20 | 北京鼎普科技股份有限公司 | Real-time threat information correlation method, device and system based on Flink |
| CN112765109A (en) * | 2021-01-20 | 2021-05-07 | 商客通尚景科技（上海）股份有限公司 | Queue type data storage analysis method and system |
| CN113067835A (en) * | 2021-04-14 | 2021-07-02 | 华能国际电力股份有限公司 | Integrated self-adaptive collapse index processing system |
| CN113489716A (en) * | 2021-07-02 | 2021-10-08 | 南京联成科技发展股份有限公司 | Threat information data correlation analysis system based on centralized management and control |
| CN113691498A (en) * | 2021-07-23 | 2021-11-23 | 全球能源互联网研究院有限公司 | Electric power internet of things terminal safety state evaluation method and device and storage medium |
| CN114666148A (en) * | 2022-03-31 | 2022-06-24 | 深信服科技股份有限公司 | Risk assessment method and device and related equipment |
| CN115442279A (en) * | 2022-09-02 | 2022-12-06 | 杭州安恒信息技术股份有限公司 | Method, device and equipment for positioning warning source and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101610174A (en) * | 2009-07-24 | 2009-12-23 | 深圳市永达电子股份有限公司 | A kind of log correlation analysis system and method |
| CN108039959A (en) * | 2017-11-29 | 2018-05-15 | 深信服科技股份有限公司 | Situation Awareness method, system and the relevant apparatus of a kind of data |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108460278A (en) * | 2018-02-13 | 2018-08-28 | 北京奇安信科技有限公司 | A kind of threat information processing method and device |
| US20190036958A1 (en) * | 2017-07-26 | 2019-01-31 | Barracuda Networks, Inc. | Method and apparatus for generating cyber security threat index |
| CN109861995A (en) * | 2019-01-17 | 2019-06-07 | 安徽谛听信息科技有限公司 | A kind of safe big data intelligent analysis method of cyberspace, computer-readable medium |
-
2020
- 2020-07-07 CN CN202010644148.4A patent/CN111770106A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101610174A (en) * | 2009-07-24 | 2009-12-23 | 深圳市永达电子股份有限公司 | A kind of log correlation analysis system and method |
| US20190036958A1 (en) * | 2017-07-26 | 2019-01-31 | Barracuda Networks, Inc. | Method and apparatus for generating cyber security threat index |
| CN108039959A (en) * | 2017-11-29 | 2018-05-15 | 深信服科技股份有限公司 | Situation Awareness method, system and the relevant apparatus of a kind of data |
| CN108427711A (en) * | 2018-01-31 | 2018-08-21 | 北京三快在线科技有限公司 | Real-time data warehouse, real-time data processing method, electronic equipment and storage medium |
| CN108460278A (en) * | 2018-02-13 | 2018-08-28 | 北京奇安信科技有限公司 | A kind of threat information processing method and device |
| CN109861995A (en) * | 2019-01-17 | 2019-06-07 | 安徽谛听信息科技有限公司 | A kind of safe big data intelligent analysis method of cyberspace, computer-readable medium |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112685394A (en) * | 2020-12-25 | 2021-04-20 | 北京鼎普科技股份有限公司 | Real-time threat information correlation method, device and system based on Flink |
| CN112765109A (en) * | 2021-01-20 | 2021-05-07 | 商客通尚景科技（上海）股份有限公司 | Queue type data storage analysis method and system |
| CN113067835A (en) * | 2021-04-14 | 2021-07-02 | 华能国际电力股份有限公司 | Integrated self-adaptive collapse index processing system |
| CN113067835B (en) * | 2021-04-14 | 2022-07-15 | 华能国际电力股份有限公司 | Integrated self-adaptive collapse index processing system |
| CN113489716A (en) * | 2021-07-02 | 2021-10-08 | 南京联成科技发展股份有限公司 | Threat information data correlation analysis system based on centralized management and control |
| CN113691498A (en) * | 2021-07-23 | 2021-11-23 | 全球能源互联网研究院有限公司 | Electric power internet of things terminal safety state evaluation method and device and storage medium |
| CN113691498B (en) * | 2021-07-23 | 2023-03-14 | 全球能源互联网研究院有限公司 | Electric power internet of things terminal safety state evaluation method and device and storage medium |
| CN114666148A (en) * | 2022-03-31 | 2022-06-24 | 深信服科技股份有限公司 | Risk assessment method and device and related equipment |
| CN114666148B (en) * | 2022-03-31 | 2024-02-23 | 深信服科技股份有限公司 | Risk assessment method and device and related equipment |
| CN115442279A (en) * | 2022-09-02 | 2022-12-06 | 杭州安恒信息技术股份有限公司 | Method, device and equipment for positioning warning source and storage medium |
| CN115442279B (en) * | 2022-09-02 | 2024-04-26 | 杭州安恒信息技术股份有限公司 | Alarm source positioning method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111770106A (en) | Method, device, system, electronic device and storage medium for data threat analysis | |
| US8549645B2 (en) | System and method for detection of denial of service attacks | |
| CN111079104A (en) | Authority control method, device, equipment and storage medium | |
| Almulla et al. | A state-of-the-art review of cloud forensics | |
| CN111786950B (en) | Network security monitoring method, device, equipment and medium based on situation awareness | |
| US9584541B1 (en) | Cyber threat identification and analytics apparatuses, methods and systems | |
| US20140280075A1 (en) | Multidimension clusters for data partitioning | |
| EP3101580B1 (en) | Website information extraction device, system, website information extraction method, and website information extraction program | |
| CN110210213B (en) | Method and device for filtering malicious sample, storage medium and electronic device | |
| CN105138709A (en) | Remote evidence taking system based on physical memory analysis | |
| CN112632129B (en) | Code stream data management method, device and storage medium | |
| CN112073437B (en) | Multi-dimensional security threat event analysis method, device, equipment and storage medium | |
| CN110782374A (en) | Electronic evidence obtaining method and system based on block chain | |
| CN107733725B (en) | Safety early warning method, device, equipment and storage medium | |
| CN112738040A (en) | Network security threat detection method, system and device based on DNS log | |
| US9992209B1 (en) | System and method for characterizing security entities in a computing environment | |
| CN105577670A (en) | Warning system of database-hit attack | |
| US10142359B1 (en) | System and method for identifying security entities in a computing environment | |
| CN111079138A (en) | Abnormal access detection method and device, electronic equipment and readable storage medium | |
| CN112165451A (en) | APT attack analysis method, system and server | |
| Hemdan et al. | Spark-based log data analysis for reconstruction of cybercrime events in cloud environment | |
| US10237287B1 (en) | System and method for detecting a malicious activity in a computing environment | |
| CN115827379A (en) | Abnormal process detection method, device, equipment and medium | |
| Avdoshin et al. | Deep web users deanonimization system | |
| CN115102785A (en) | Automatic tracing system and method for network attack |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201013 |