CN114143173B - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114143173B CN114143173B CN202210113307.7A CN202210113307A CN114143173B CN 114143173 B CN114143173 B CN 114143173B CN 202210113307 A CN202210113307 A CN 202210113307A CN 114143173 B CN114143173 B CN 114143173B
- Authority
- CN
- China
- Prior art keywords
- data
- threat
- target
- processing
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Alarm Systems (AREA)
Abstract
The embodiment of the invention provides a data processing method, a data processing device, data processing equipment and a storage medium. The data processing method comprises the following steps: acquiring original safety data; performing target processing on the original safety data to obtain threat data, wherein the target processing is used for screening the threat data from the original safety data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data; and identifying target threat data in the original safety data according to the alarm data and the threat event data. In the process, aiming at a large amount of original safety data, the data amount can be reduced and the information contained in the data can be increased through target processing. On the basis, compared with the method for directly identifying the target threat data from a large amount of original safety data, the method for identifying the target threat data from the threat data after target processing is easier and more efficient.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
With the arrival of the internet plus, network data is increased explosively, data such as logs, flow and alarms generated by various devices are gathered in a data warehouse, the data volume of the data is huge, and if the data based on the data warehouse is needed to perform processing such as identification and analysis of security threats, the efficiency is low.
Disclosure of Invention
To solve the problems in the prior art, embodiments of the present invention provide a data processing method, apparatus, device, and storage medium.
Specifically, the embodiment of the invention provides the following technical scheme:
in a first aspect, an embodiment of the present invention provides a data processing method. The data processing method comprises the following steps:
acquiring original security data;
performing target processing on the original safety data to obtain threat data, wherein the target processing is used for screening the threat data from the original safety data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data;
and identifying target threat data in the original safety data according to the alarm data and the threat event data.
Further, the target processing is performed on the original security data to obtain threat data, and the method comprises the following steps:
performing threat level mapping processing on data with a threat level field in the original safety data to obtain the alarm data; the threat level mapping process is used for mapping the threat level of the original security data into a uniform identifier;
merging the data with the target field in the alarm data to obtain the threat event data; the target field is used for representing that the data is threat event data.
Further, before performing threat level mapping processing on the data with the threat level field in the original security data, at least one of the following is further included:
carrying out format conversion on the original security data; the format of the data after the format conversion processing is the same; or the like, or a combination thereof,
adding information of the threatened target object in the original security data.
Further, before the merging the data with the target field in the alarm data, the method further includes:
and performing noise reduction processing on the alarm data according to the target field.
Further, merging the data with the target field in the alarm data to obtain the threat event data, including:
acquiring at least one preset safety rule;
classifying data with target fields in the alarm data according to the at least one preset safety rule to obtain at least one type of threat event data;
and aiming at any type of the threat event data, merging the threat event data according to a preset merging rule.
Further, the preset merging rule includes a preset time range and a preset attribute, and the merging processing of any type of the threat event data according to the preset merging rule includes:
and merging the threat event data of which the time stamp is in the preset time range and comprises the preset attribute in any type of the threat event data.
Further, after the merging processing is performed according to the preset merging rule, the method further includes:
updating or adding an alarm field of the merged threat event data to obtain the threat event data, wherein the alarm field comprises at least one of the following items: threat level, attributes, and treatment recommendations.
Further, after identifying the target threat data in the original security data according to the alarm data and the threat event data, the method further comprises the following steps:
threat data of the same threatened target object included in the target threat data is obtained.
In a second aspect, an embodiment of the present invention further provides a data processing apparatus, configured to sense a network security situation, where the apparatus includes:
the acquisition module is used for acquiring original safety data;
the processing module is used for carrying out target processing on the original safety data to obtain threat data, and the target processing is used for screening the threat data from the original safety data; the threat data includes: alarm data and threat event data; the alert data includes the threat event data;
the processing module is further used for identifying target threat data in the original safety data according to the alarm data and the threat event data.
In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the data processing method according to the first aspect when executing the program.
In a fourth aspect, the present invention further provides a non-transitory computer readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the data processing method according to the first aspect.
In a fifth aspect, the present invention further provides a computer program product, on which executable instructions are stored, and when executed by a processor, the instructions cause the processor to implement the steps of the data processing method according to the first aspect.
According to the data processing method, the data processing device, the data processing equipment and the storage medium, threat data are obtained by performing target processing on original safety data, and the target processing is used for screening the threat data from the original safety data; the threat data includes: alarm data and threat event data; the alarm data comprises the threat event data, i.e. the data of the original security data, the alarm data and the threat event with reduced data volume after processing by the target form a data "pyramid" structure. In the process, aiming at a large amount of original safety data, the data amount can be reduced through target processing. On the basis, compared with the method for directly identifying the target threat data from a large amount of original safety data, the method is easier and more efficient to identify the target threat data from the alarm data and the threat event data after target processing.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is an application scenario diagram of an embodiment of a data processing method provided in the present invention;
FIG. 2 is a flowchart illustrating an embodiment of a data processing method according to the present invention;
FIG. 3 is a schematic flow chart illustrating a data processing method according to another embodiment of the present invention;
fig. 4 is a schematic diagram of data flow of an embodiment of the data processing method provided in the present invention;
FIG. 5 is a schematic diagram of an embodiment of a data processing apparatus according to the present invention;
fig. 6 is a schematic structural diagram of an embodiment of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
Situation awareness is an environment-based, dynamic and overall security risk awareness technology, and is a way to improve the capabilities of discovery, identification, comprehension, analysis and response handling of security threats from a global perspective on the basis of security big data. With the arrival of the internet +, network data is increased explosively, and data such as logs, flow and alarms generated by various devices are gathered in a data warehouse, so that the data volume is huge. When the network security situation awareness is carried out by using data in the existing data warehouse, because the data volume is huge, the threatening data is not easy to be found, namely, the processing efficiency is low.
The method provided by the embodiment of the invention can be applied to a network security situation perception scene, firstly, a plurality of original security data are subjected to target processing to obtain a plurality of first alarm data, and then threat data are identified according to the plurality of first alarm data. For example, the network security posture of a website of a business entity is perceived.
Fig. 1 is an application scenario diagram provided in an embodiment of the present invention, and optionally, as shown in fig. 1, the application scenario includes at least one user equipment 11, an electronic device 12, and a server 13; the user device 11 may be a terminal device, such as a desktop computer, a notebook computer, a tablet, a mobile phone, etc., where a client of the user is located. The user device 11 may be a client of a website of a certain utility, i.e. a producer of network data. The server 13 may be the server side of the utility's website, and also the provider of the original security data.
The electronic device 12 is mainly used for acquiring original security data from the server 13. The electronic device 13 is further configured to perform target processing on the original security data to obtain threat data, where the target processing is configured to screen out the threat data from the original security data; the threat data includes: alarm data and threat event data; the alarm data comprises threat event data; and identifying target threat data in the original safety data according to the alarm data and the threat event data. The user equipment 11, the electronic device 12, and the server 13 may be connected to each other through a network, for example, a communication network such as 4G, 5G, or Wireless Fidelity (WIFI).
The method provided by the present invention can be implemented by the electronic device 12 such as a processor executing corresponding software codes, or can be implemented by the electronic device 12 executing corresponding software codes and performing data interaction with the server 13, for example, the server executes a part of operations to control the electronic device to execute the cache data updating method.
The following embodiments are all described with electronic devices as the executing bodies. The technical solution of the present invention is described in detail with specific embodiments in conjunction with fig. 2-5. These several specific embodiments may be combined with each other below, and details of the same or similar concepts or processes may not be repeated in some embodiments.
Fig. 2 is a schematic flowchart of an embodiment of a data processing method according to the present invention. As shown in fig. 2, a method provided in an embodiment of the present invention includes:
and step 101, acquiring original safety data.
Specifically, the original security data mainly refers to various flow data, alarm data, log data, and the like. The log mainly comprises a login log, an access log and the like. The original security data can be sourced from various data providers or data probes.
For example, for a business unit website, traffic data, alarm data, and log data for the business unit website may be obtained from a data detector.
For another example, for a communication Application (APP), traffic data, alarm data, and log data of the communication APP may be obtained from an application manufacturer.
102, performing target processing on the original safety data to obtain threat data, wherein the target processing is used for screening the threat data from the original safety data; the threat data includes: alarm data and threat event data; the alert data includes threat event data.
Specifically, the threat data is screened from the original security data, and may include: alarm data and threat event data. The alarm data mainly comprises data of various threat levels and data with potential threats. The threat data and the raw security data may be stored in a server, such as a full text search engine (Lucene) based search (ElasticSearch) server, which maintains a basis for subsequent traceability. The above-described target process may be a simplified screening process that reduces the amount of data. After the target processing, the effect of reducing the data volume can be achieved.
Moreover, the alarm data includes threat event data, that is, the threat event data is data screened from the alarm data, for example, data extracted according to some specific fields, data with a greater threat degree, so that the threat event data is further reduced compared with the alarm data, a data pyramid structure is obtained, that is, valuable target threat data is easily obtained through data layer-by-layer screening.
For example, for original security data of a website of a certain event entity, threat data obtained after target processing may include alarm data of each threat level, traffic data with potential threats, and threat event data. The alarm data and the threat event data not only reduce the data volume relative to the original safety data, but also directly represent threat information, such as information of threat objects with threats and potential threats.
And 103, identifying target threat data in the original safety data according to the alarm data and the threat event data.
Specifically, by analyzing or further monitoring alarm data and threat event data, valuable target threat data in the original security data can be identified. The identification method can be various situation awareness methods.
For example, for a website of a certain enterprise, the alarm data and the threat event data obtained in step 102 may be analyzed by using a situation awareness method to identify target threat data in the original security data.
According to the data processing method provided by the embodiment of the invention, threat data are obtained by performing target processing on original safety data, wherein the target processing is used for screening the threat data from the original safety data; the threat data includes: alarm data and threat event data; the alarm data comprises the threat event data, and the original security data, the alarm data and the threat event data form a pyramid structure of data. In the process, aiming at a large amount of original safety data, the data amount can be reduced through target processing. On the basis, compared with the method for directly identifying the target threat data from a large amount of original safety data, the method is easier and more efficient to identify the target threat data from the alarm data and the threat event data after target processing.
Alternatively, as shown in fig. 3, step 102 may be implemented as follows:
carrying out threat level mapping processing on data with a target field in original safety data to obtain alarm data; the threat level mapping process is used for mapping the threat level of the original security data into a uniform identifier;
merging the data with the target field in the alarm data to obtain threat event data; the target field is used for representing that the data is threat event data.
In particular, for heterogeneous original security data from multiple vendors, adaptation problems may exist due to different sources. The threat levels of the original safety data are mapped, so that the original safety data of different sources can be classified and identified according to the same threat level rule, and meanwhile, the data without the threat levels are filtered. Data with target fields in the alarm data can be upgraded into threat event data, and the target fields are threat indicator ioc fields and the like; the data volume can be further reduced by carrying out merging processing, wherein the merging processing refers to merging the data with the target fields in the alarm data together according to a certain merging rule to obtain threat event data. For example, for a plurality of original security data of a website of a certain event unit, threat level mapping processing may be performed first, so that the alarm data has a uniform threat level identifier, and invalid data is removed; and then merging the data with the target field in the alarm data, namely the data which is raised to be the threat event, so as to further reduce the data volume and obtain the threat event data, and optionally enriching the data by increasing the information of the associated field of the data and the like, so that the value of the data can be more greatly exerted during the subsequent association analysis.
For example, for original security data of a website of a certain business entity, the threat level rules of all manufacturers are different for the threat level of vulnerability data in the original security data, the grading of manufacturer 1 is common, and the grading of manufacturer 2 is less harmful. In the operation of updating the threat level of the original security data, the threat level identification is carried out in a numbering mode. And the manufacturer 1 is rated as common, and the manufacturer 2 is rated as less-harmful vulnerability data, and the threat level of the vulnerability data is updated to be 0. In the subsequent operation, each alarm data carries the updated threat level to be transferred.
In the embodiment, the threat level mapping processing and the merging processing are sequentially performed on the original security data, so that different types of processing and classification can be conveniently performed on the original security data, threat data in the original security data can be found favorably, and a basis is provided for subsequently identifying target threat data.
In one implementation, before performing the threat level mapping process on the data with the threat level field in the original security data, at least one of the following may be further included:
carrying out format conversion on the original security data; the format of the data after the format conversion processing is the same; or the like, or, alternatively,
information of the target object under threat is added to the original security data.
Specifically, the original security data from different vendors has differences in protocol and field. The types of the original security data may include system log (syslog) data, JavaScript object notation (json) data from hypertext Transfer Protocol over secure browser Layer (https), file class data, and the like. By performing format conversion on a plurality of original safety data, the format of the data after the format conversion can be the same, so that the alarm data can be conveniently circulated in the same data format.
For example, the original security data for a website of a certain event unit comprises three formats of syslog data, json data and file class data. And in the process of target processing, format conversion is carried out on the original safety data, syslog data and file data are converted into json data, and the formats of the processed alarm data are json data.
Specifically, the information of the threatened target object refers to the threatened target object when some original security data is threat data, and the unit, industry, area information and the like of the threatened target object. In practical application, information of the threatened target object may be added to all original security data, or information of the corresponding threatened target object may be added to part of the original security data.
For example, the original security data of a certain business website includes vulnerability data only including information of the vulnerability itself. In the process of target processing, information such as the unit, industry and threatened area information of the website to which the related threat is attached can be added to the vulnerability data.
In the embodiment, the threat level of the original security data is updated, so that the original security data can be identified according to the same threat level rule, the original security data can be classified and identified according to the threat level, the fast circulation of data is facilitated, the data without the threat level is filtered, and the data volume is reduced. The format conversion is carried out on the original safety data, so that the original safety data can be circulated and stored in the same data format, the universality and the flexibility of the original safety data can be improved, and the complexity of data circulation and storage can be reduced. The information of the threatened target object is added to the original safety data, and when certain data is subsequently reported, rectified and the like, the collection of related information is facilitated, and the working efficiency can be improved. When the threat level of the original safety data is updated and the format of the original safety data is converted and the information of the threatened target object is added, the original safety data is stored and circulated by carrying the detailed information of the threatened target object in the same format and the threat level identified by the same threat level rule, so that the data processing efficiency can be improved, and the threatened data can be conveniently found.
In an embodiment, before performing merging processing on data with a target field in alarm data, the method further includes:
and performing noise reduction processing on the alarm data according to the target field.
Specifically, the noise reduction processing refers to removing meaningless data, such as the alarm data, which cannot be analyzed. The method for performing noise reduction processing may be to determine whether the alarm data has a key target field, and if a certain alarm data does not have a target field, the alarm data cannot be analyzed, belongs to meaningless data, and needs to be removed.
For example, for the alarm data of a website of a certain event entity, whether the alarm data has a threat indicator (ioc) field or not can be determined, and if the ioc field is provided, the alarm data can be used for analysis and can be reserved as meaningful data. If the ioc field is not provided, the alarm data cannot be used for analysis, and is meaningless data and can be removed.
At this time, in the process of performing target processing on the alarm data, the noise reduction processing can greatly reduce the data amount needing attention, and improve the pertinence and the data processing efficiency. In addition, the merging process after the noise reduction process can reduce the data amount without affecting the threat data discovery. Based on this, the efficiency of threat data discovery, as well as the discovery capabilities, may be improved.
Optionally, the merging processing of the data with the target field in the alarm data may be specifically implemented by the following manner:
acquiring at least one preset safety rule;
classifying data with target fields in the alarm data according to at least one preset safety rule to obtain at least one type of threat event data;
and aiming at any type of threat event data, merging the threat event data according to a preset merging rule.
Specifically, the acquired preset security rule mainly includes information such as a rule name and a start rule. The preset safety rule can be a rule formed by performing sum (and) or (or) and not (not) logic processing among a plurality of fields. The preset safety rules can be set according to the characteristics of the attention points and the data, and default safety rules can be preset. For example, the rules are: the data type (data-type) is Structured Query Language (SQL) injection and the website domain name is 127.0.0.1. And the data matched with the rule in the alarm data of a certain service unit is vulnerability data. All vulnerability data matched to the rule are in the same class. And (4) transferring the threat event data belonging to the vulnerability data category to the next step, and merging according to a preset merging rule.
Optionally, the preset merge rule may include a preset time range and a preset attribute. Merging any type of threat event data according to a preset merging rule, and realizing the merging processing by the following mode:
and merging the threat event data of which the time stamp is in a preset time range and which comprises preset attributes in any type of threat event data.
Specifically, for at least one type of threat event data classified according to a preset security rule, threat event data included in any type of threat event data are merged according to a preset time range and a preset attribute, so that the data volume of the threat event data obtained by merging each type of threat event data is reduced. In practical applications, the duration of the preset time range and the preset attribute can be set according to the data processing requirements. For example, the preset merge rule includes a preset time range of 1 hour, and the preset attribute is a Uniform Resource Locator (URL). And for vulnerability alarm data of a website of a certain business unit, merging the data with the same URL into threat event data within 1 hour.
The preset merging rules are utilized to carry out merging processing, and data merging can be carried out on the basis of alarm data classification. At the moment, the data volume can be reduced, the data processing efficiency is improved, the identification of threat data in the alarm data can not be influenced, and the identification capability of the threat data is ensured.
In the process of merging the data with the target field in the alarm data, the alarm data are classified according to the preset safety rules, the alarm data can be reasonably classified according to the data processing requirements, the data can be better managed, and the data utilization rate and the threat data discovery capability are improved. Meanwhile, the merging is carried out through the preset merging rule, so that the data volume can be reduced, and the data processing efficiency is improved.
In an embodiment, after the merging processing is performed according to the preset merging rule, the following steps may be further performed:
updating or adding an alarm field of the merged threat event data to obtain the threat event data, wherein the alarm field comprises at least one of the following items: threat level, attributes, and treatment recommendations.
Specifically, in the merged threat event data, the data of one threat event may be merged by a plurality of data before merging. The threat level of the merged data of one threat event can be updated according to the original threat levels of a plurality of alarm data composing the threat event. The attribute of the merged data of one threat event can be determined according to the original attributes of a plurality of alarm data composing the merged data. The merged disposal suggestion of one threat event data may be updated according to the original disposal suggestions of the plurality of alarm data composing the merged disposal suggestion, or a new disposal suggestion may be added according to the updated threat level.
For example, for a vulnerability datum merged by a website of a certain business unit, the vulnerability datum is merged by 3 data. The first data has a threat level of 0, an attribute of event _ level, and a handling recommendation of security. The second data has a threat level of 1, an attribute of event _ level, and a treatment recommendation of repair. The third data has a threat level of 2, an attribute of event _ level, and a treatment recommendation of repair. When the alarm field is updated or increased, the threat level of the combined vulnerability data is updated to 2, the attribute is updated to explicit _ event _ level, the disposal suggestion is updated to repair, and the disposal suggestion for checking the threatened target object is increased.
In the above embodiment, the alarm field is updated or added for the data of the merged threat event, so that the target threat data is convenient to be re-determined, the efficiency of discovering the target threat data can be effectively improved, and the target threat data is convenient to be disposed.
In one embodiment, after identifying the target threat data in the original security data based on the alert data and the data of the threat event, the method further comprises:
threat data for the same target object that is compromised included in the target threat data is obtained.
Specifically, obtaining threat data of the same threatened target object included in the target threat data means to summarize corresponding target threat data of the same threatened target object. The target threat data is mainly attack chains, information of the threatened target object, severity, influence range and the like.
For example, for a website of a business entity, the authority to acquire a server for the website, information attacked by a script, severity of possible server paralysis, influence on the business and the region of the business entity, and the like may be aggregated.
In the embodiment, by summarizing the target threat data of the same target object, valuable target threat data can be screened, and powerful support is provided for subsequent rectification and corresponding legal program intervention.
Illustratively, as shown in fig. 3, the data processing method of the present embodiment includes the following steps:
step 1, obtaining an original database comprising original safety data. In this step, a plurality of raw security data is streamed from the data provider to the acquisition module.
And 2, carrying out threat level mapping processing on the data with the threat level field in the original security data. The threat level is identified from low to high using numbers of 0, 1, 2, 3, 4. In this step, the raw security data is streamed from the acquisition module to the threat level update module.
And 3, converting the format of the original safety data. For example, after conversion, the original security data formats in the original database are all json formats. In this step, the raw security data flows from the threat level update module to the format conversion module.
And 4, adding information of the threatened target object related to the self service to the plurality of original safety data. At this time, a plurality of second alarm data and corresponding alarm databases are obtained. Such as the affiliated unit, affiliated industry, affected area information, etc. of the affected website. In this step, a plurality of raw security data is streamed from the format conversion module to the add field module to add the relevant information. Subsequently, the alarm data is transferred from the add field module to the data storage module. In addition, the original security data can be directly transferred to the data storage module from a data provider, and then transferred to the threat level updating module, the format conversion module and the field adding module from the data storage module, so as to complete data processing.
And 5, performing noise reduction processing on the alarm data in the alarm database. For example, if data missing the ioc field cannot be analyzed, the data is discarded. At this time, the alarm data is transferred from the data storage module to the data denoising and merging module, and the alarm data completes the processing of the data in step 5 and the subsequent steps 6, 7 and 8 in the data denoising and merging module.
And 6, acquiring a preset safety rule, and classifying data with target fields in the alarm data in the alarm database according to the preset safety rule.
And 7, merging the threat event data of each type. And merging the data of the same URL in each hour into the data of one threat event.
And 8, updating or increasing alarm fields of the merged data of the threat events to obtain a plurality of threat events and corresponding event databases. At this point, data of the threat event is stored in the server.
And 9, identifying target threat data in the original safety data according to an event database comprising data of the threat event. In this step, data of threat events are streamed from the server to the situation awareness platform to identify target threat data therein.
And step 10, summarizing the attacked situation of the same attack target.
In the data processing method provided by this embodiment, a plurality of first alarm data are obtained by performing target processing on a plurality of original security data. In the process, aiming at a large amount of original safety data, the data amount can be reduced and the information contained in the data can be increased through target processing. On the basis, compared with the method for directly identifying the threat data from a large amount of original safety data, the method for identifying the threat data from the plurality of first alarm data processed by the target is easier and more efficient.
The data processing device provided by the invention is described below, and the data processing device described below and the data processing method described above can be referred to correspondingly.
Fig. 5 is a schematic structural diagram of an embodiment of a data processing apparatus according to the present invention. As shown in fig. 5, the data processing apparatus provided in this embodiment includes:
an obtaining module 210, configured to obtain a plurality of original security data;
a processing module 220, configured to perform target processing on the original security data to obtain threat data, where the target processing is used to screen out threat data from the original security data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data;
the processing module 220 is further configured to identify target threat data in the original security data according to the alarm data and the threat event data.
The data processing apparatus provided in this embodiment performs target processing on a plurality of original security data to obtain a plurality of first alarm data. In the process, aiming at a large amount of original safety data, the data amount can be reduced through target processing. On the basis, compared with the method for directly identifying the threat data from a large amount of original safety data, the method for identifying the threat data from the plurality of first alarm data processed by the target is easier and more efficient.
Optionally, the processing module 220 is specifically configured to:
performing threat level mapping processing on data with a threat level field in the original safety data to obtain the alarm data; the threat level mapping process is used for mapping the threat level of the original security data into a uniform identifier;
merging the data with the target field in the alarm data to obtain the data of the threat event; the target field is used for representing that the data is threat event data.
Optionally, the processing module 220 is further configured to perform at least one of the following operations:
carrying out format conversion on the original safety data to obtain the alarm data; the format of the data after the format conversion processing is the same; or the like, or a combination thereof,
information of a target object subject to threat is added to the original security data.
Optionally, the processing module 220 is specifically configured to:
and performing noise reduction processing on the alarm data according to the target field.
Optionally, the processing module 220 is specifically configured to:
acquiring at least one preset safety rule;
classifying data with target fields in the alarm data according to the at least one preset safety rule to obtain at least one type of threat event data;
and aiming at any type of the threat event data, combining the threat event data according to a preset merging rule.
Optionally, the preset merge rule includes a preset time range and a preset attribute, and the processing module 220 is specifically configured to:
and merging the threat event data of which the time stamp is in the preset time range and comprises the preset attribute in any type of the threat event data.
Optionally, the processing module 220 is further configured to:
updating or adding an alarm field of the merged threat event data to obtain the threat event data, wherein the alarm field comprises at least one of the following items: threat level, attributes, and treatment recommendations.
Optionally, the processing module 220 is further configured to:
and acquiring threat data of the same threatened target object included in the target threat data.
The apparatus according to the embodiment of the present invention is configured to perform the method according to any of the foregoing method embodiments, and the implementation principle and technical effects are similar, which are not described herein again.
An example is as follows:
fig. 6 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 6: a processor (processor)810, a communication Interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication Interface 820 and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to perform the following method: acquiring original security data; performing target processing on the original safety data to obtain threat data, wherein the target processing is used for screening the threat data from the original safety data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data; and identifying target threat data in the original safety data according to the alarm data and the threat event data.
In addition, the logic instructions in the memory 830 can be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention or a part thereof which substantially contributes to the prior art may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, where the computer program is implemented to, when executed by a processor, perform the data processing method provided by the foregoing embodiments, and the method includes: acquiring original safety data; performing target processing on the original safety data to obtain threat data, wherein the target processing is used for screening the threat data from the original safety data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data; and identifying target threat data in the original safety data according to the alarm data and the threat event data.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on the understanding, the above technical solutions substantially or otherwise contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A data processing method, comprising:
acquiring original security data;
performing target processing on the original safety data to obtain threat data, wherein the target processing is used for screening the threat data from the original safety data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data;
identifying target threat data in the original safety data according to the alarm data and the threat event data;
the target processing is performed on the original safety data to obtain threat data, and the method comprises the following steps:
carrying out threat level mapping processing on data with a threat level field in the original safety data to obtain the alarm data; the threat level mapping process is used for mapping the threat level of the original security data into a uniform identifier;
merging the data with the target field in the alarm data to obtain the threat event data; the target field is used for representing that the data is threat event data.
2. The data processing method of claim 1, wherein before performing the threat level mapping process on the data with the threat level field in the original security data, at least one of the following is further included:
carrying out format conversion on the original safety data; the format of the data after the format conversion processing is the same; or the like, or, alternatively,
adding information of the threatened target object in the original security data.
3. The method according to claim 2, wherein before the merging the data with the target field in the alarm data, further comprising:
and performing noise reduction processing on the alarm data according to the target field.
4. The method of claim 2, wherein merging data having a target field in the alert data to obtain the threat event data comprises:
acquiring at least one preset safety rule;
classifying data with target fields in the alarm data according to the at least one preset safety rule to obtain at least one type of threat event data;
and aiming at any type of the threat event data, merging the threat event data according to a preset merging rule.
5. The method of claim 4, wherein the preset merge rule includes a preset time range and a preset attribute, and merging any type of the threat event data according to the preset merge rule includes:
and merging the threat event data of which the time stamp is in the preset time range and comprises the preset attribute in any type of the threat event data.
6. The method according to claim 4 or 5, wherein after the merging process according to the preset merging rule, the method further comprises:
updating or adding an alarm field of the merged threat event data to obtain the threat event data, wherein the alarm field comprises at least one of the following items: threat level, attributes, and treatment recommendations.
7. The method of claim 1 or 2, further comprising, after identifying target threat data in the raw security data based on the alert data and threat event data:
threat data of the same threatened target object included in the target threat data is obtained.
8. A data processing apparatus for network security posture awareness, comprising:
the acquisition module is used for acquiring original security data;
the processing module is used for carrying out target processing on the original safety data to obtain threat data, and the target processing is used for screening the threat data from the original safety data; the threat data comprises: alarm data and threat event data; the alert data includes the threat event data;
the processing module is further used for identifying target threat data in the original safety data according to the alarm data and the threat event data;
wherein the processing module is specifically configured to:
carrying out threat level mapping processing on data with a threat level field in the original safety data to obtain the alarm data; the threat level mapping process is used for mapping the threat level of the original security data into a uniform identifier;
merging the data with the target field in the alarm data to obtain the threat event data; the target field is used for representing that the data is threat event data.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and running on the processor, characterized in that the steps of the data processing method according to any of claims 1 to 7 are implemented when the program is executed by the processor.
10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the data processing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210113307.7A CN114143173B (en) | 2022-01-30 | 2022-01-30 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210113307.7A CN114143173B (en) | 2022-01-30 | 2022-01-30 | Data processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114143173A CN114143173A (en) | 2022-03-04 |
| CN114143173B true CN114143173B (en) | 2022-07-15 |
Family
ID=80381862
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210113307.7A Active CN114143173B (en) | 2022-01-30 | 2022-01-30 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114143173B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114884801A (en) * | 2022-06-09 | 2022-08-09 | 奇安信科技集团股份有限公司 | Alarm method, alarm device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105681286A (en) * | 2015-12-31 | 2016-06-15 | 中电长城网际系统应用有限公司 | Association analysis method and association analysis system |
| CN113542253A (en) * | 2021-07-12 | 2021-10-22 | 杭州安恒信息技术股份有限公司 | Network flow detection method, device, equipment and medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108460278B (en) * | 2018-02-13 | 2020-07-14 | 奇安信科技集团股份有限公司 | Threat information processing method and device |
| US11206265B2 (en) * | 2019-04-30 | 2021-12-21 | Infoblox Inc. | Smart whitelisting for DNS security |
| US11748488B2 (en) * | 2019-12-24 | 2023-09-05 | Sixgill Ltd. | Information security risk management |
| CN113515433B (en) * | 2021-07-28 | 2023-08-15 | 中移（杭州）信息技术有限公司 | Alarm log processing method, device, equipment and storage medium |
-
2022
- 2022-01-30 CN CN202210113307.7A patent/CN114143173B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105681286A (en) * | 2015-12-31 | 2016-06-15 | 中电长城网际系统应用有限公司 | Association analysis method and association analysis system |
| CN113542253A (en) * | 2021-07-12 | 2021-10-22 | 杭州安恒信息技术股份有限公司 | Network flow detection method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114143173A (en) | 2022-03-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11870795B1 (en) | Identifying attack behavior based on scripting language activity | |
| US11032304B2 (en) | Ontology based persistent attack campaign detection | |
| US10404731B2 (en) | Method and device for detecting website attack | |
| US9350747B2 (en) | Methods and systems for malware analysis | |
| CN107480277B (en) | Method and device for collecting website logs | |
| US10795991B1 (en) | Enterprise search | |
| US20140156711A1 (en) | Asset model import connector | |
| CN103685575A (en) | Website security monitoring method based on cloud architecture | |
| CN111290916A (en) | Big data monitoring method, device and equipment and computer readable storage medium | |
| CN111740868B (en) | Alarm data processing method and device and storage medium | |
| US20190281064A1 (en) | System and method for restricting access to web resources | |
| US11178160B2 (en) | Detecting and mitigating leaked cloud authorization keys | |
| CN112000992B (en) | Data leakage prevention protection method and device, computer readable medium and electronic equipment | |
| CN114143173B (en) | Data processing method, device, equipment and storage medium | |
| CN112347165A (en) | Log processing method and device, server and computer readable storage medium | |
| US7392430B2 (en) | System and program product for checking a health of a computer system | |
| US20220417263A1 (en) | Browser extension for cybersecurity threat intelligence and response | |
| CN106339372B (en) | Method and device for optimizing search engine | |
| CN116991675A (en) | Abnormal access monitoring method and device, computer equipment and storage medium | |
| US11734297B1 (en) | Monitoring platform job integration in computer analytics system | |
| JP7140268B2 (en) | WARNING DEVICE, CONTROL METHOD AND PROGRAM | |
| CN115714662A (en) | Processing method of multi-source data, alarm analysis method, device and equipment | |
| CN117978450A (en) | Security detection method, device, equipment and storage medium | |
| KR101503463B1 (en) | Internal monitoring of vessels using the weighted average | |
| WO2023205349A1 (en) | Method, apparatus, system, and non-transitory computer readable medium for identifying and prioritizing network security events |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |