CN111988341A - Data processing method, device, computer system and storage medium - Google Patents
Data processing method, device, computer system and storage medium Download PDFInfo
- Publication number
- CN111988341A CN111988341A CN202010950419.9A CN202010950419A CN111988341A CN 111988341 A CN111988341 A CN 111988341A CN 202010950419 A CN202010950419 A CN 202010950419A CN 111988341 A CN111988341 A CN 111988341A
- Authority
- CN
- China
- Prior art keywords
- entity object
- entity
- file
- objects
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data processing method, including: acquiring original data related to safety; extracting a plurality of entity objects from raw data; processing the entity objects by using a threat intelligence data set to obtain respective label characteristics of each entity object in the entity objects, wherein the label characteristics are used for representing the security attributes and/or malicious attributes of the entity objects; and determining threat information of each entity object according to the label characteristics of each entity object. The present disclosure provides a data processing apparatus, a computer system, and a storage medium.
Description
Technical Field
The present disclosure relates to the field of network security technologies, and in particular, to a data processing method, apparatus, computer system, and storage medium.
Background
With the rapid development of computer and internet technologies, network security issues are becoming the focus of people's attention. The attack modes of attackers are more and more diversified, and malicious information may exist in network data streams in various forms, so that the judgment and marking of the malicious information in the network data are very important.
At present, in the related technology, the information of malicious families and attack groups is not completely mastered, the judgment and marking of malicious information in network data need to depend on manual review, the judgment efficiency is low, and misjudgment exists.
Disclosure of Invention
In view of the above, the present disclosure provides a data processing method, apparatus, computer system, and storage medium.
One aspect of the present disclosure provides a data processing method, including: acquiring original data related to safety; extracting a plurality of entity objects from the raw data; processing the entity objects by using a threat intelligence data set to obtain respective label characteristics of each entity object in the entity objects, wherein the label characteristics are used for representing the security attributes and/or malicious attributes of the entity objects; and determining the threat information of each entity object according to the label characteristic of each entity object.
According to an embodiment of the present disclosure, the threat intelligence data set includes a plurality of knowledge bases; processing the plurality of physical objects with a threat intelligence dataset comprises: processing the entity object by utilizing at least one knowledge base in a plurality of knowledge bases aiming at each entity object in the plurality of entity objects to obtain a processing result of each knowledge base in the at least one knowledge base, wherein each knowledge base comprises a plurality of entity objects marked with tag characteristics; and determining the label characteristics of the entity object according to the processing result of each knowledge base.
According to an embodiment of the present disclosure, processing the entity object using at least one of a plurality of knowledge bases includes: for each of the at least one knowledge bases, determining whether a target entity object identical to the entity object is included in the knowledge base; and if the target entity object which is the same as the entity object is determined to be included in the knowledge base, marking the entity object by using the label characteristic of the target entity object.
According to an embodiment of the present disclosure, determining threat information of each physical object according to the tag feature of each physical object includes: and aiming at each entity object, processing the label characteristics of the entity object by using a network model to obtain threat information of the entity object.
According to an embodiment of the present disclosure, the entity object includes a file; the method further comprises the following steps: extracting the characteristics of the file to obtain the static characteristics and the dynamic characteristics of the file; running the file by using a sandbox to obtain the behavior characteristics of the file; and determining threat information of the file according to at least one of static characteristics, dynamic characteristics, behavior characteristics and tag characteristics of the file.
According to an embodiment of the present disclosure, the method further comprises: integrating the entity objects processed by the threat intelligence data set; and associating the integrated entity objects according to the label characteristics of the entity objects to obtain an entity object relation data set.
According to an embodiment of the present disclosure, the method further comprises: acquiring a new entity object; processing the new entity object by using the entity object relation data set to obtain the label characteristic of the new entity object; and determining threat information of the entity object according to the label characteristics of the new entity object.
According to an embodiment of the present disclosure, the entity object includes at least one of a file, a domain name, an IP, and a web address; the at least one knowledge base comprises at least one of a white name list base, a black name list base, a recorded domain name base, a credit file base, a lost host base and a credit IP base; the threat information includes at least one of a malicious type, attacker information, and means of attack.
Another aspect of the present disclosure provides a data processing apparatus including: the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring original data related to safety; a first extraction module, configured to extract a plurality of entity objects from the raw data; the first processing module is used for processing the entity objects by utilizing a threat intelligence data set to obtain respective label characteristics of each entity object in the entity objects, wherein the label characteristics are used for representing the security attributes and/or malicious attributes of the entity objects; and the determining module is used for determining the threat information of each entity object according to the label characteristics of each entity object.
Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.
Another aspect of the disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.
Another aspect of the present disclosure provides a computer system comprising: one or more processors; storage means for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described above.
According to the embodiment of the disclosure, a plurality of entity objects are extracted from original data, the entity objects are processed by using a threat intelligence data set to obtain respective label characteristics of each entity object, and threat information of each entity object is determined according to the label characteristics of each entity object. Because the entity object is processed by utilizing the threat information data set and the malicious information of the entity object is labeled, the technical problems of low judgment efficiency and low accuracy rate caused by dependence on manual audit on judgment and marking of the malicious information in the related technology are at least partially solved, and the technical effects of improving the judgment and marking efficiency and accuracy rate of the malicious information are further achieved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
fig. 1 schematically shows an exemplary system architecture to which the data processing method of the embodiments of the present disclosure may be applied;
FIG. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a method for processing a plurality of physical objects with a threat intelligence data set, in accordance with an embodiment of the disclosure;
FIG. 4 schematically illustrates a flow diagram of a method for processing an entity object utilizing at least one of a plurality of knowledge bases, in accordance with an embodiment of the present disclosure;
FIG. 5 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure;
FIG. 6 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure;
FIG. 7 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure;
FIG. 8 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and
FIG. 9 schematically shows a block diagram of a computer system according to an embodiment of the disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
APT (Advanced Persistent thread, APT for short) is also called "target-specific" attack, which is a new type of attack that is organized, has a specific target, and has a very long duration. APT attacks are continuous, malicious information is layered endlessly, and security manufacturers also continuously utilize various tools to detect network attacks so as to track and locate the malicious information.
An operation platform for detecting network attacks is generally arranged in a security manufacturer, and the platform can be operated with various tools for detecting network attacks, for example, a sandbox is adopted to detect the network attacks, a webpage crawler is used to detect the network attacks, and sandbox operation logs, network crawler logs and the like are used to detect the network attacks. However, this requires manual intervention, such as manually setting sandbox operating conditions, manually writing crawler code, etc., for different situations. And the detection result needs further analysis and audit manually, so that the efficiency is low, and the condition of misjudgment or missed judgment may occur.
Based on this, embodiments of the present disclosure provide a data processing method and apparatus. The method comprises the following steps: acquiring original data related to safety; extracting a plurality of entity objects from raw data; processing the entity objects by using a threat intelligence data set to obtain respective label characteristics of each entity object in the entity objects, wherein the label characteristics are used for representing the security attributes and/or malicious attributes of the entity objects; and determining threat information of each entity object according to the label characteristics of each entity object.
Fig. 1 schematically illustrates an exemplary system architecture 100 to which the data processing method of the disclosed embodiments may be applied. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, a system architecture 100 according to this embodiment may include an electronic device 101, a threat intelligence data set-based detection platform 102, and a manual operation tagging platform 103.
After the electronic device 101 acquires the original data related to the security, it may first perform data extraction to obtain a plurality of IOC (index of compliance, IOC for short) entity objects. The IOC entity object may be of various types, such as a file, an IP, a domain name, a HOST, a web page address, etc., wherein the file may be represented by MD5(Message Digest Algorithm) of the file, and the web page address may include a main domain name HOST, a URL (Uniform Resource Locator), a URI (Uniform Resource Identifier). The IOC data may then be passed through the detection platform 102 and the manual operation tagging platform 103, respectively, for detection of malicious information. Specifically, the raw data may be a network data stream, a file, a message, and the like.
The threat intelligence data may be data obtained by processing, analyzing, determining and/or marking source data of various dimensions. The source data of different dimensions may include open source information data, business information data (such as attack events issued by various security vendors), operation data of security products (such as attack alarm information), data output by the file deep analysis engine (such as file types), determination results of multi-virus engine scanning, and manual operation data (such as crawler data and sandbox logs).
The method comprises the steps of extracting safe entity objects such as IP, network addresses, domain names, files and the like aiming at source data, intercepting useful attribute information such as security attributes, malicious attributes, context attributes and the like by the entity objects through field comparison and extraction, judging whether the objects are malicious information according to the attributes of the entity objects so as to calibrate the malicious information, and storing the entity objects in a database after malicious judgment and marking to form a threat information data set.
The manual operation marking platform 103 runs a web crawler tool, a sandbox tool, and the like, and can be used for assisting the detection of the detection platform 102, and also can adjust and correct the detection result of the detection platform 102. The manual operation marking platform 103 mainly analyzes malicious information and performs manual judgment and marking by using webpage crawler data, detection logs, network activity information obtained by sandbox operation and the like.
The IOC entity object is detected by the detection platform 102 and/or the manual operation marking platform 103, and a label is marked in each detection link according to a detection result, wherein the label can represent the security attribute and/or the malicious attribute of the IOC object. For example, a file is detected by a white list database, a blacklist database, and/or a reputation file database in the detection platform 102, and if the file is a malicious file, the file may be marked with a file type, an attack means, and the like, for example, a label may be marked with a black file, used in a certain vulnerability attack, and the like. After the file is detected by the manual operation marking platform 103, if a malicious sample generated by the file for a certain malicious family is detected, a label of the known malicious family by an attacker can be marked.
It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by the electronic device 101. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may be generally disposed in the electronic device 101. The data processing method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster that is different from the electronic device 101 and is capable of communicating with the electronic device 101. Accordingly, the data processing apparatus provided in the embodiments of the present disclosure may also be disposed in a server or a server cluster different from the electronic device 101 and capable of communicating with the electronic device 101.
It should be understood that the electronic device 101, the detection platform 102, the knowledge base in the detection platform 102, the manually operated marking platform 103, and the number of detection tools in the manually operated marking platform 103 in fig. 1 are merely illustrative. There may be any number of electronic devices 101, detection platforms 102, knowledge bases, manual operation tagging platforms 103, and detection tools, as desired for an implementation.
Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S201 to S204.
In operation S201, raw data related to security is acquired.
In operation S202, a plurality of entity objects are extracted from raw data.
According to an embodiment of the present disclosure, the raw data may be, for example, a network data stream, a service data packet, and the like. The IOC entity objects, which may include MD5, IP, domain name, HOST, URL, etc., are extracted from the raw data. The IOC data can also be subjected to normalization processing, denoising processing, de-duplication processing, field completion and the like to normalize the data.
In operation S203, the threat intelligence data set is used to process the plurality of entity objects, and a tag characteristic of each entity object in the plurality of entity objects is obtained, where the tag characteristic is used to characterize a security attribute and/or a malicious attribute of the entity object.
According to an embodiment of the present disclosure, the threat intelligence data set includes a plurality of knowledge bases, such as a whitelist database, a blacklist database, a lost host database, a reputation file base, a reputation IP base, a docket domain name base, and the like.
The threat intelligence data is obtained by processing multi-dimensional source data, and the form of the threat intelligence data can be a composite expression of some attributes. Attributes to which threat intelligence data relates may include: observable behavior characteristics: phenomena such as network congestion, system corruption, etc.; threat characteristic indexes are as follows: whether the asset information is really attacked by the threat can be judged by looking at the characteristics; description of security events: the method comprises the following steps of (1) behavior of malicious attack, a damaged target, what weak point is utilized, influence and consequence, killer chain information and the like; the attack intention describes: why the attack, including the attacker's characteristics, intent, organization to which it belongs, etc., is to be launched; vulnerability characterization: actions taken against the attacked system; tracing information: the specific information of the attack initiator includes the characteristics of organization, national mailbox, account number and the like.
In the above attribute features, an attribute characterizing a malicious feature or a security feature of the entity object itself may be used as a tag feature, and a feature characterizing a context association relationship of the entity object may be used as a context attribute.
According to the embodiment of the disclosure, the multidimensional source data can be derived from open source information data, commercial information data, operation data of a safety product, data output by a file depth analysis engine, judgment results of multi-virus engine scanning, webpage crawler data, detection logs, network activity information obtained by sandbox operation and the like, the source data of multiple links are covered, data under a full-link scene is automatically obtained, and the multidimensional degree, the breadth and the depth of a data source are ensured.
According to this disclosed embodiment, utilize threat intelligence data set to detect IOC entity object, because threat intelligence data set handles the source data of multidimension and obtains, compare in traditional utilization single source data and detect IOC entity object, guaranteed the breadth and the degree of depth of data, and then guaranteed the continuation and the new freshness of source data, improved the comprehensive and the accuracy of the analysis and detection of IOC entity object.
According to an embodiment of the disclosure, each entity object is processed by using at least one knowledge base in the threat intelligence data set, for example, the entity object may be compared with entity objects in the knowledge base, and if matching is successful, a tag attribute of an entity object in the knowledge base matching the current entity object may be assigned to the current entity object.
According to the embodiment of the disclosure, the context attribute of the entity object matching the current entity object in the knowledge base can be further allocated to the current entity object to enrich the attribute information of the current entity object.
In operation S204, threat information of each physical object is determined according to the tag feature of each physical object.
According to the embodiment of the disclosure, after each entity object is judged by at least one knowledge base and a manual operation marking platform, a plurality of label attributes are marked. For example, a certain IOC object is marked with a tag 1, a tag 2, and a tag 3 after being processed by the knowledge base 1, marked with a tag 4, a tag 5, and a tag 6 after being processed by the knowledge base 3, and marked with a tag 7 and a tag 8 after being processed by the manual operation marking platform. Some attribute tags may be included in the tags, and malicious tags may also be included in the tags.
According to the embodiment of the disclosure, the malicious tags can be mainly focused, and the malicious tag with the strongest representative is selected to mark the entity object. For example, the tag 1, the tag 4, and the tag 7 may be selected to mark the entity object, and threat information such as attacker information, attack means, and malicious type may be determined according to the malicious tags, which may specifically include an attack group name, a country, a mailbox, an account, vulnerability information utilized by an attacker, and an attack tool adopted by an attacker.
According to an embodiment of the present disclosure, operation S204 may include: and processing the label characteristics of the entity object by using a network model aiming at each entity object to obtain the threat information of the entity object. The network model can be a classification model obtained based on machine learning clustering algorithm training, the input of the classification model can be the label characteristics of the entity object, and the output can be threat information such as attack groups, attack means, malicious types and the like to which the label characteristics belong.
According to the embodiment of the disclosure, after the labels returned by different knowledge bases are clustered through a machine learning clustering algorithm, the attribution information and the family group information of the data labels can be automatically judged.
The machine learning clustering algorithm can be a random forest algorithm or an IBk (k nearest neighbor classification) algorithm is introduced on the basis of the random forest algorithm, so that the weakness that false alarm is caused by the fact that the random forest falls into a local optimal solution can be overcome, the IBk algorithm trains a training set used by the random forest algorithm to generate a IBk model, and the model supports an unknown sample to search three most similar samples from the training set. By combining the random forest algorithm and the IBk algorithm, the APT sample and the malicious family sample are detected with high precision and high accuracy. Other machine learning classification clustering algorithms can also be used, and the type of the machine learning algorithm is not limited by the disclosure.
According to the embodiment of the disclosure, a plurality of entity objects are extracted from original data, a threat intelligence data set is utilized to process the entity objects, respective label characteristics of each entity object are obtained, and threat information of each entity object is determined according to the label characteristics of each entity object. Because the entity object is processed by utilizing the threat intelligence data set, the malicious information of the entity object is labeled, and the efficiency and the accuracy of judging and marking the malicious information can be improved.
Furthermore, according to the tag characteristics of the entity object, threat information such as attack group information, malicious family information, attack methods and the like of malicious information can be quickly positioned, and quick tracking and accurate positioning can be realized for attacks of organizations such as APT and the like.
FIG. 3 schematically illustrates a flow chart of a method for processing a plurality of physical objects with a threat intelligence data set, in accordance with an embodiment of the disclosure.
As shown in fig. 3, operation S203 includes operations S301 to S302.
In operation S301, the entity object is processed by using at least one of the knowledge bases to obtain a processing result of each of the at least one knowledge base, wherein each of the knowledge bases includes a plurality of entity objects marked with tag features.
According to the embodiment of the disclosure, different studying and judging processes are required for different IOC objects. For example, if the IOC object is a file, then a white list library, a reputation file library, and a manual operations study flow need to be performed. If the IOC object is an IP, the study and judgment of a white list library, an IP reputation library and a manual operation marking process are needed. If the IOC object is a domain name, then a white name list library, a lost host library, a recorded domain name library and a manual operation research and judgment process are required. If the IOC object is a URL, a white list library and a manual operation research and judgment process are required.
According to the embodiment of the disclosure, for each type of IOC, the judging results in different knowledge bases may be the same or different, and after the judgment of at least one knowledge base, the manual operation marking platform can be used for adjustment, so that the marking accuracy is improved.
For example, for a certain URL object, after the detection of the white list library, no malicious tag is marked, and after the detection of the manual operation marking platform, a malicious download tag is marked, it is determined that the URL is malicious.
In operation S302, tag characteristics of the entity object are determined according to the processing result of each knowledge base.
According to the embodiment of the disclosure, for each IOC, each time the IOC is processed by one knowledge base, rich label features can be marked, the same or different labels can be obtained by processing different knowledge bases, the same labels can be deduplicated, and the different labels are integrated to obtain the complete label features of the entity object.
FIG. 4 schematically illustrates a flow chart of a method for processing an entity object using at least one of a plurality of knowledge bases in accordance with an embodiment of the disclosure.
As shown in fig. 4, operation S301 includes operations S401 to S402.
In operation S401, it is determined whether a target entity object identical to the entity object is included in the knowledge base for each of the at least one knowledge base.
In operation S402, if it is determined that the same target entity object as the entity object is included in the knowledge base, the entity object is tagged with the tag feature of the target entity object.
For example, a domain name is compared with a host domain name in the lost host library, and if the domain name exists in the lost host library, the label attribute of the domain name in the lost host library is marked to the current domain name.
According to an embodiment of the present disclosure, operation S301 further includes: and for each knowledge base in at least one knowledge base, performing information completion on the context characteristics of the current entity object according to the context characteristics of the entity object in the knowledge base, and enriching the label information generated by the label, thereby ensuring the stability and accuracy of the output of the labeled information.
Fig. 5 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure.
As shown in fig. 5, operations S501 to S503 are included.
In operation S501, feature extraction is performed on the file to obtain static features and dynamic features of the file.
In operation S502, the file is run using the sandbox to obtain the behavior characteristics of the file.
According to the embodiment of the disclosure, for the IOC object of the file type, processing can also be performed by using a deep file parsing engine and sandbox operation.
Specifically, the static analysis and the dynamic analysis can be performed on the file through the deep file parsing engine to obtain the static characteristics and the dynamic characteristics of the file. Static characteristics may include file name, file size, time to first occurrence of detection, etc. The dynamic characteristics may be some information of File operation and mapped information, including maximum and minimum stream, stream type, compiler type, PDB (Program Database File) length, string length, number of dictionary elements, array size, associated File information, etc.
Specifically, the file is run through the sandbox, so that characteristics of a submission area, sandbox network information, file release information and the like can be obtained.
In operation S503, threat information of the file is determined according to at least one of a static feature, a dynamic feature, a behavior feature, and a tag feature of the file.
According to the embodiment of the disclosure, comprehensive judgment is performed on static characteristics and dynamic characteristics obtained based on a depth file analysis engine, behavior characteristics obtained based on sandbox operation and label characteristics obtained based on a threat intelligence data set. All context information required by file calibration can be acquired, and final label attributes and context attributes are obtained after multi-dimensional statistical association.
Fig. 6 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure.
As shown in fig. 6, operations S601 to S602 are included.
In operation S601, the entity objects processed by the threat intelligence data set are integrated.
In operation S602, the integrated entity objects are associated according to the tag characteristics of the entity objects, so as to obtain an entity object relationship data set.
According to the embodiment of the present disclosure, after the IOC data is subjected to the detection and tagging processing by the detection platform 102 and/or the manual operation tagging platform 103, the IOC data may be integrated, for example, the same IOC data is subjected to deduplication processing, field completion, standardization processing, and the like, and the same IOC data is integrated together. Related IOCs can also be aggregated according to the label characteristics, for example, IOC data belonging to the same malicious family are related together, for example, an IP can be related to a file or URL, etc., and finally a new IOC data set with the association relationship is formed.
Fig. 7 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure.
As shown in fig. 7, operations S701 to S703 are included.
In operation S701, a new entity object is acquired.
In operation S702, the new entity object is processed by using the entity object relationship data set to obtain a tag feature of the new entity object.
In operation S703, threat information of the entity object is determined according to the tag feature of the new entity object.
According to the embodiment of the disclosure, the IOC data set with the association relationship can provide a uniform API interface, associate different service analysis systems and provide different query processing services. Therefore, when a new IOC entity object is obtained, threat information such as a malicious type, an attack means, a malicious family to which the IOC entity object belongs can be inquired and obtained through the API interface.
Fig. 8 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 8, the data processing apparatus 800 includes a first acquisition module 801, a first extraction module 802, a first processing module 803, and a determination module 804.
The first obtaining module 801 is used for obtaining raw data related to security.
The first extraction module 802 is used to extract a plurality of entity objects from the raw data.
The first processing module 803 is configured to process the plurality of entity objects by using the threat intelligence data set, to obtain a tag feature of each entity object in the plurality of entity objects, where the tag feature is used to characterize a security attribute and/or a malicious attribute of the entity object.
The first determining module 804 is configured to determine threat information of each entity object according to the tag feature of each entity object.
The first processing module 803 includes a processing unit and a determination unit according to an embodiment of the present disclosure.
The processing unit is used for processing the entity object by using at least one of the knowledge bases to obtain a processing result of each knowledge base in the at least one knowledge base, wherein each knowledge base comprises a plurality of entity objects marked with tag characteristics.
The determining unit is used for determining the label characteristics of the entity object according to the processing result of each knowledge base.
According to an embodiment of the present disclosure, a processing unit includes a determination subunit and a marking subunit.
The determining subunit is configured to determine, for each of the at least one knowledge base, whether a target entity object identical to the entity object is included in the knowledge base.
The labeling subunit is configured to label the entity object with a label feature of the target entity object if it is determined that the target entity object identical to the entity object is included in the knowledge base.
According to the embodiment of the present disclosure, the first determining module 804 is configured to, for each entity object, process the tag feature of the entity object by using a network model, so as to obtain threat information of the entity object.
According to an embodiment of the present disclosure, the data processing apparatus 800 further includes a second extraction module, a second processing module, and a second determination module.
The second extraction module is used for extracting the characteristics of the file to obtain the static characteristics and the dynamic characteristics of the file.
The second processing module is used for operating the file by using the sandbox to obtain the behavior characteristics of the file.
The second determining module is used for determining the threat information of the file according to at least one of the static characteristic, the dynamic characteristic, the behavior characteristic and the label characteristic of the file.
According to an embodiment of the present disclosure, the data processing apparatus 800 further includes an integration module and an association module.
The integration module is used for integrating the entity objects processed by the threat intelligence data set.
And the association module is used for associating the integrated entity object according to the label characteristics of the entity object to obtain an entity object relationship data set.
According to the embodiment of the present disclosure, the data processing apparatus 800 further includes a second obtaining module, a third processing module, and a third determining module.
The second acquisition module is used for acquiring a new entity object.
And the third processing module is used for processing the new entity object by using the entity object relation data set to obtain the label characteristic of the new entity object.
And the third determining module is used for determining the threat information of the entity object according to the label characteristic of the new entity object.
According to an embodiment of the present disclosure, the entity object includes at least one of a file, a domain name, an IP, and a web address. The at least one knowledge base includes at least one of a white list base, a black list base, a docket domain name base, a reputation file base, a lost host base, and a reputation IP base. The threat information includes at least one of a malicious type, attacker information, and means of attack.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, any plurality of the first obtaining module 801, the first extracting module 802, the first processing module 803, and the determining module 804 may be combined and implemented in one module/unit/sub-unit, or any one of the modules/units/sub-units may be split into a plurality of modules/units/sub-units. Alternatively, at least part of the functionality of one or more of these modules/units/sub-units may be combined with at least part of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to an embodiment of the present disclosure, at least one of the first obtaining module 801, the first extracting module 802, the first processing module 803, and the determining module 804 may be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware, and firmware, or by a suitable combination of any of them. Alternatively, at least one of the first obtaining module 801, the first extracting module 802, the first processing module 803, and the determining module 804 may be at least partially implemented as a computer program module, which when executed, may perform a corresponding function.
It should be noted that, the data processing apparatus portion in the embodiment of the present disclosure corresponds to the data processing method portion in the embodiment of the present disclosure, and the description of the data processing apparatus portion specifically refers to the data processing method portion, which is not described herein again.
FIG. 9 schematically shows a block diagram of a computer system suitable for implementing the above described method according to an embodiment of the present disclosure. The computer system illustrated in FIG. 9 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.
As shown in fig. 9, a computer system 900 according to an embodiment of the present disclosure includes a processor 901 which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage section 908 into a Random Access Memory (RAM) 903. Processor 901 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 901 may also include on-board memory for caching purposes. The processor 901 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.
In the RAM 903, various programs and data necessary for the operation of the system 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. The processor 901 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 902 and/or the RAM 903. Note that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 909, and/or installed from the removable medium 911. The computer program, when executed by the processor 901, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 902 and/or the RAM 903 described above and/or one or more memories other than the ROM 902 and the RAM 903.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (11)
1. A method of data processing, comprising:
acquiring original data related to safety;
extracting a plurality of entity objects from the raw data;
processing the entity objects by using a threat intelligence data set to obtain respective label characteristics of each entity object in the entity objects, wherein the label characteristics are used for representing the security attributes and/or malicious attributes of the entity objects; and
and determining the threat information of each entity object according to the label characteristics of each entity object.
2. The method of claim 1, wherein the threat intelligence data set comprises a plurality of knowledge bases;
processing the plurality of physical objects with a threat intelligence dataset comprises: for each of the plurality of entity objects,
processing the entity object by using at least one knowledge base in a plurality of knowledge bases to obtain a processing result of each knowledge base in the at least one knowledge base, wherein each knowledge base comprises a plurality of entity objects marked with tag characteristics; and
and determining the label characteristics of the entity object according to the processing result of each knowledge base.
3. The method of claim 2, wherein processing the entity object with at least one of a plurality of repositories includes: for each of the at least one knowledge base,
determining whether a target entity object identical to the entity object is included in the knowledge base; and
and if the knowledge base comprises the target entity object which is the same as the entity object, marking the entity object by using the label characteristic of the target entity object.
4. The method of claim 1, wherein determining threat information for the each physical object based on the tag characteristics of the each physical object comprises: for each of the entity objects, a function is performed,
and processing the label characteristics of the entity object by using a network model to obtain threat information of the entity object.
5. The method of claim 1 or 2, wherein the entity object comprises a file;
the method further comprises the following steps:
extracting the characteristics of the file to obtain the static characteristics and the dynamic characteristics of the file;
running the file by using a sandbox to obtain the behavior characteristics of the file; and
and determining threat information of the file according to at least one of the static characteristic, the dynamic characteristic, the behavior characteristic and the label characteristic of the file.
6. The method of claim 1, further comprising:
integrating the entity objects processed by the threat intelligence data set; and
and associating the integrated entity objects according to the label characteristics of the entity objects to obtain an entity object relation data set.
7. The method of claim 6, further comprising:
acquiring a new entity object;
processing the new entity object by using the entity object relation data set to obtain the label characteristic of the new entity object; and
and determining the threat information of the entity object according to the label characteristics of the new entity object.
8. The method of any one of claims 1-7, wherein:
the entity object comprises at least one of a file, a domain name, an IP and a webpage address;
the at least one knowledge base comprises at least one of a white name list base, a black name list base, a recorded domain name base, a credit file base, a lost host base and a credit IP base;
the threat information includes at least one of a malicious type, attacker information, and means of attack.
9. A data processing apparatus comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring original data related to safety;
a first extraction module, configured to extract a plurality of entity objects from the raw data;
the first processing module is used for processing the entity objects by utilizing a threat intelligence data set to obtain respective label characteristics of each entity object in the entity objects, wherein the label characteristics are used for representing the security attributes and/or malicious attributes of the entity objects; and
and the determining module is used for determining the threat information of each entity object according to the label characteristics of each entity object.
10. A computer system, comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-8.
11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010950419.9A CN111988341B (en) | 2020-09-10 | 2020-09-10 | Data processing method, device, computer system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010950419.9A CN111988341B (en) | 2020-09-10 | 2020-09-10 | Data processing method, device, computer system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111988341A true CN111988341A (en) | 2020-11-24 |
| CN111988341B CN111988341B (en) | 2022-08-02 |
Family
ID=73451038
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010950419.9A Active CN111988341B (en) | 2020-09-10 | 2020-09-10 | Data processing method, device, computer system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111988341B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113919514A (en) * | 2021-12-09 | 2022-01-11 | 北京微步在线科技有限公司 | Sample data acquisition method and device based on threat intelligence |
| CN113992371A (en) * | 2021-10-18 | 2022-01-28 | 安天科技集团股份有限公司 | Method and device for generating threat tag of flow log and electronic equipment |
| CN114844691A (en) * | 2022-04-20 | 2022-08-02 | 安天科技集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115037523A (en) * | 2022-05-17 | 2022-09-09 | 浙江工业大学 | APT detection method for heterogeneous terminal log fusion |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160080417A1 (en) * | 2014-09-14 | 2016-03-17 | Sophos Limited | Labeling computing objects for improved threat detection |
| US20170250997A1 (en) * | 2016-02-29 | 2017-08-31 | Palo Alto Networks, Inc. | Alerting and tagging using a malware analysis platform for threat intelligence made actionable |
| CN108460278A (en) * | 2018-02-13 | 2018-08-28 | 北京奇安信科技有限公司 | A kind of threat information processing method and device |
| CN110198303A (en) * | 2019-04-26 | 2019-09-03 | 北京奇安信科技有限公司 | Threaten the generation method and device, storage medium, electronic device of information |
| CN110659493A (en) * | 2019-09-25 | 2020-01-07 | 哈尔滨安天科技集团股份有限公司 | Method and device for generating threat alarm mode, electronic equipment and storage medium |
| CN110875920A (en) * | 2018-12-24 | 2020-03-10 | 哈尔滨安天科技集团股份有限公司 | Network threat analysis method and device, electronic equipment and storage medium |
| CN110955893A (en) * | 2019-11-22 | 2020-04-03 | 杭州安恒信息技术股份有限公司 | Malicious file threat analysis platform and malicious file threat analysis method |
| US20200162484A1 (en) * | 2017-05-22 | 2020-05-21 | Leap In Value S.L. | A computer-implemented method, a system and a computer program for identifying malicious URI data items |
| CN111800395A (en) * | 2020-06-18 | 2020-10-20 | 云南电网有限责任公司信息中心 | Threat information defense method and system |
-
2020
- 2020-09-10 CN CN202010950419.9A patent/CN111988341B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160080417A1 (en) * | 2014-09-14 | 2016-03-17 | Sophos Limited | Labeling computing objects for improved threat detection |
| US20170250997A1 (en) * | 2016-02-29 | 2017-08-31 | Palo Alto Networks, Inc. | Alerting and tagging using a malware analysis platform for threat intelligence made actionable |
| US20200162484A1 (en) * | 2017-05-22 | 2020-05-21 | Leap In Value S.L. | A computer-implemented method, a system and a computer program for identifying malicious URI data items |
| CN108460278A (en) * | 2018-02-13 | 2018-08-28 | 北京奇安信科技有限公司 | A kind of threat information processing method and device |
| CN110875920A (en) * | 2018-12-24 | 2020-03-10 | 哈尔滨安天科技集团股份有限公司 | Network threat analysis method and device, electronic equipment and storage medium |
| CN110198303A (en) * | 2019-04-26 | 2019-09-03 | 北京奇安信科技有限公司 | Threaten the generation method and device, storage medium, electronic device of information |
| CN110659493A (en) * | 2019-09-25 | 2020-01-07 | 哈尔滨安天科技集团股份有限公司 | Method and device for generating threat alarm mode, electronic equipment and storage medium |
| CN110955893A (en) * | 2019-11-22 | 2020-04-03 | 杭州安恒信息技术股份有限公司 | Malicious file threat analysis platform and malicious file threat analysis method |
| CN111800395A (en) * | 2020-06-18 | 2020-10-20 | 云南电网有限责任公司信息中心 | Threat information defense method and system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113992371A (en) * | 2021-10-18 | 2022-01-28 | 安天科技集团股份有限公司 | Method and device for generating threat tag of flow log and electronic equipment |
| CN113992371B (en) * | 2021-10-18 | 2023-08-18 | 安天科技集团股份有限公司 | Threat label generation method and device for traffic log and electronic equipment |
| CN113919514A (en) * | 2021-12-09 | 2022-01-11 | 北京微步在线科技有限公司 | Sample data acquisition method and device based on threat intelligence |
| CN114844691A (en) * | 2022-04-20 | 2022-08-02 | 安天科技集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114844691B (en) * | 2022-04-20 | 2023-07-14 | 安天科技集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN115037523A (en) * | 2022-05-17 | 2022-09-09 | 浙江工业大学 | APT detection method for heterogeneous terminal log fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111988341B (en) | 2022-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111988341B (en) | Data processing method, device, computer system and storage medium | |
| CN108763928B (en) | Open source software vulnerability analysis method and device and storage medium | |
| Piplai et al. | Creating cybersecurity knowledge graphs from malware after action reports | |
| CN108092962B (en) | Malicious URL detection method and device | |
| Namanya et al. | Similarity hash based scoring of portable executable files for efficient malware detection in IoT | |
| US11126720B2 (en) | System and method for automated machine-learning, zero-day malware detection | |
| US9300682B2 (en) | Composite analysis of executable content across enterprise network | |
| US9237161B2 (en) | Malware detection and identification | |
| US9953162B2 (en) | Rapid malware inspection of mobile applications | |
| CN110177114B (en) | Network security threat indicator identification method, equipment, device and computer readable storage medium | |
| US20200380125A1 (en) | Method for Detecting Libraries in Program Binaries | |
| US10097569B2 (en) | System and method for tracking malware route and behavior for defending against cyberattacks | |
| EP3346664B1 (en) | Binary search of byte sequences using inverted indices | |
| WO2020000743A1 (en) | Webshell detection method and related device | |
| Upchurch et al. | Variant: a malware similarity testing framework | |
| CN108446559B (en) | APT organization identification method and device | |
| CN112019519B (en) | Method and device for detecting threat degree of network security information and electronic device | |
| CN109829304B (en) | Virus detection method and device | |
| CN103279710A (en) | Method and system for detecting malicious codes of Internet information system | |
| CN111104579A (en) | Identification method and device for public network assets and storage medium | |
| US20230254326A1 (en) | System and Method for Information Gain for Malware Detection | |
| US10437995B2 (en) | Systems and methods for inference of malware labels in a graph database | |
| CN115309796A (en) | Similarity query method, database updating method, device and system | |
| US11321453B2 (en) | Method and system for detecting and classifying malware based on families | |
| CN116975865A (en) | Malicious Office document detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: Room 332, 3 / F, Building 102, 28 xinjiekouwei street, Xicheng District, Beijing 100088 Applicant after: Qianxin Technology Group Co.,Ltd. Applicant after: Qianxin Wangshen information technology (Beijing) Co.,Ltd. Address before: Room 332, 3 / F, Building 102, 28 xinjiekouwei street, Xicheng District, Beijing 100088 Applicant before: Qianxin Technology Group Co.,Ltd. Applicant before: LEGENDSEC INFORMATION TECHNOLOGY (BEIJING) Inc. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |