CN104166680A - Parallel vulnerability mining method based on open source library and text mining - Google Patents

Abstract

The invention relates to a parallel vulnerability mining method based on an open source library and test mining, and belongs to the technical field of computer information safety. The parallel vulnerability mining method comprises the steps that vulnerability data are obtained from the open source library and pre-processed, a vulnerability set is extracted, text vectorization is conducted, the threshold is calculated and parallel vulnerabilities are discovered. The parallel vulnerability mining method has the advantages that on the basis of the open source library, relevant vulnerability information in the same attack mode is extracted, and therefore potential parallel relationships between the vulnerabilities can be analyzed conveniently; text description information of the vulnerabilities are vectorized, and therefore a computer system can conduct intelligent processing on vulnerability recording data conveniently; the method differs from query on the basis of keyword matching in that the similarity between the vulnerabilities is studied according to the threshold obtained through a training set; the parallel relationships between the vulnerabilities can be calculated, so that when it is found that one vulnerability is utilized, the other parallel vulnerabilities are made up rapidly, therefore, the vulnerability of a whole network is made up, the defense capacity is enhanced and great significance for information safety is achieved.

Description

A kind of parallel bug excavation method based on increase income storehouse and text mining

Technical field

The invention belongs to computer information safety technique field.Particularly, the present invention relates to a kind of parallel bug excavation method based on increase income storehouse and text mining.

Background technology

Along with the constantly universal and develop rapidly of Internet, people use and depend on internet more and more.Meanwhile, the economic loss producing because of internet information safety problem significantly improves, and the harm causing also obviously increases.Be present in the leak in various software and operating system, for hackers start network attack, steal user profile, even destroy industrial infrastructure condition is provided.The effective ways of potential safety hazard in guarding network will not be studied in countries in the world surplus energy for this reason.

Since 1996, information security expert just started internet attack mode to conclude and sum up, and attempted to understand network attack from higher level.And at this wherein, CAPEC (the Common AttackPattern Enumeration and Classification) project that the Ze Shiyou U.S. Department of Homeland Security of extensively being approved (United States Department of Homeland Security, DHS) supports.In CAPEC, each attack mode is designated unique ID, and following information is provided: attack mode description, attack step, precondition (as needed satisfied technical conditions before attack such as the platform relying on and software version), postcondition (as the administrator right of obtaining after attack etc.), attack instance, relevant CWE (Common Weakness Enumeration) fragility and CVE (Common Vulnerabilities and Exposures) leak etc.CAPEC has not only set forth the detail of network attack mode, has also indicated contacting between it and CWE, CVE.CWE is a fragility set of describing in software.For each fragility, it provides following information: fragility description, applicable platform, the result causing, example and relevant CVE leak.A CWE fragility correspondence many relevant CVE leaks.

The parallel leak the present invention relates to (Parallel Vulnerabilities), refers to and has identical attack basic condition, can reach identical attack object and effect but utilize the leak combination of different attack paths.In brief, the approach that parallel leak comes analytical attack to utilize by the method for multipath, thus improve probability and the coverage rate of successful defending against network attacks.

What have close ties with parallel leak is attack graph: attack graph has been described from assailant or defender's visual angle and how to be utilized the leak existing system to reach the object of attacking or taking precautions against; System management can be assessed by attack graph the security of their system, and determines to take what kind of remedial measures to take precautions against.Parallel leak is exactly to come from same a starting point in attack graph and the branch parallel relation between end point, for the leak combination of finding that these have concurrency relation, can find that a leak makes up rapidly other parallel leak while being utilized, be convenient in time to corresponding leak patch installing, and then make up the fragility of whole network, there is higher cyber-defence using value.

Summary of the invention

The object of the invention is to, based on the information bank of increasing income, extract associated fragility CWE under same attack mode, from fragility CWE, be associated with leak CVE again, then by text mining and natural language processing technique, excavate parallel leak, thereby there is higher cyber-defence using value.

Technical scheme of the present invention is:

A parallel bug excavation method based on increase income storehouse and text mining, step comprises:

1. data acquisition and pre-service:

A) from the storehouse of increasing income, obtain original attack mode information, vulnerability information and vulnerability information

B) the library information data of increasing income of obtaining are carried out to data scrubbing, comprise Uniform data format, data purification, filling disappearance attribute and remove noise data

C) data after cleaning are set up to data acquisition, and form the mapping that is identified to non-structured text information, form three set Pattern{p ₁, p ₂..., p _n, Weak{w ₁, w ₂..., w _m, Vulnerability{v ₁, v ₂..., v _s.P wherein _irepresent an attack mode, w _irepresent a fragility, v _irepresent a leak.

2. extract mapping relations and leak set

Parse attack mode p _ito fragility w _imapping relations (one-to-many), fragility w _ito leak v _imapping relations (one-to-many), then form attack mode p _ito leak v _imapping relations, thereby huge vulnerability database is concluded to different leak set according to some attack modes.

3. the textual description information of pair leak is carried out mathematical modeling

A) extract the textual description information of leak, carry out participle, extract stem, according to feature dictionary, to describing, carry out text vector, can be expressed as D _i=(w _{1, i}, w _{2, i}, w _{3, i}..., w _n,i), w wherein _n,irepresent that entry n is in document D _iin quantitative index represent.

B) according to the descriptor of text vector, calculate TF/IDF (word frequency/frequency of the falling document) value of document word, and then document can be expressed as d _i=(w _{1, i}, w _{2, i}, w _{3, i}..., w _n,i), w wherein _n, _irepresent that entry n is at document d _iin TF/IDF weighted value.

4. the leak of the leak set of extracting in pair step 2, screens according to the precondition of attack mode, postcondition and threshold value, obtains the set of parallel leak combination

Parrallel{(v _i,v _j)|i≠j,v _i∈vulnerability}

A), according to the parallel leak training set of artificial mark, calculate the threshold value (method is shown in step 4-a) of cosine similar value of parallel leak and the threshold value of the Euclidean distance of CVSS assessment mark

B) in the leak set of obtaining in step 2, according to the leak vector information getting in step 3, the cosine similar value (method is shown in step 4-a) of calculating respectively two leaks corresponding to precondition and the postcondition of attack mode

C) according to the Euclidean distance of the assessment mark of the cosine similar value of two leaks and CVSS, by two all the leak in threshold range separately differentiate for parallel leak.

Step 3-a) segmenting method described in, its step, for text flow is separated by space, obtains word sequence.Remove the character of particular meaning, replace //, ,/, be/, removal word tail number.

Step 3-b) described in, extract the method for stem, adopt Krovetz Stemmer to process text.

Step 3-b) described in, calculate the method for text TF/IDF value,

TF(t,d)＝0.5+(0.5*f(t,d))/(max{f(w,d):w∈d})

IDF(t,D)＝log(|D|)/(|{d∈D:t∈d}|)

Wherein f (t, d) is the word frequency of word t in document d, and f (w, d) is the word frequency of word w in document d, and w belongs to document d, and it can represent any one word wherein; | D| is all number of files that comprise in corpus D, | { d ∈ D:t ∈ d}| is the number of files that comprises word t in corpus D.

Step 4-a) described in, calculate the method for cosine similarity,

$\begin{array}{c}{s}_2=\mathrm{similarity}{v}_j,v_k=\frac{{\mathrm{\Σ}}_{i=1}^n{w}_{i,j}\×w_{\begin{array}{cc}i& k\end{array}}}{\sqrt{{\mathrm{\Σ}}_{i=1}^n{\left(w_{i,j}\right)}^2}\×\sqrt{{\mathrm{\Σ}}_{i=1}^n{\left(w_{\begin{array}{cc}i& k\end{array}}\right)}^2}}\\ =\frac{{\mathrm{\Σ}}_{i=1}^n\mathrm{TFIDF}(t_i,v_j)\×\mathrm{TFIDF}(t_i,v_k)}{{\sqrt{{\mathrm{\Σ}}_{i=1}\left(\mathrm{TFIDF}(t_j,v_j)\right)}}^2\×{\sqrt{{\mathrm{\Σ}}_{i=1}^n\left(\mathrm{TFIDF}(t_i,v_k)\right)}}^2}\end{array}$

V wherein _jthe description text that represents leak j, v _kthe description text that represents leak k.W _i,jrepresent that entry i is at document d _jin TF/IDF weighted value, w _i,krepresent that entry i is at document d _kin TF/IDF weighted value.

V _jthe description text that represents leak j, v _kthe description text that represents leak k; w _i,jrepresent that entry i is at document d _jin TF/IDF weighted value, w _i,krepresent that entry i is at document d _kin TF/IDF weighted value.

Advantage of the present invention mainly contains:

1. the text description field value of existing leak record data is non-structured text data, and different leaks describes source and cause its descriptor format different with word, and computing machine cannot complete the automatic Understanding to text description field automatically.The present invention carries out numerical value vectorization processing according to feature lexicon to text description field value, and indefinite text description is made clear, is convenient to computer system the intellectuality of leak record data, robotization, scale are processed.

2. between the leak of existing leak database record, only have consistent, the dangerous program height of clear and definite classification, discovery time relation successively, security attack dependence between leak record cannot embody, the present invention can excavate between leak data parallel linked model and the rule implying, for the further application of leak data provides basis.

3. existing attack mode is only included the attack mode of the network attack having occurred, and corresponding vulnerability information is single fixing, and extendability is not strong, and foresight is not strong, and the discovery of corresponding attack mode is very difficult.The present invention arrives fragility according to the attack mode of increasing income in storehouse, arrive the mapping relations of leak, parallel leak is found in recycling text mining again, to the attack mode having produced, can expand multiple leak combination, also can predict certain attack mode, for Protection of Network Security provides the foundation.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of system

Fig. 2 is the obtaining and pretreatment process figure of square frame 1001 data in Fig. 1

Fig. 3 is that in Fig. 1, square frame 1005 extracts leak set process flow diagram

Fig. 4 is square frame 1009 text vector process flow diagrams in Fig. 1

Fig. 5 is that in Fig. 1, square frame 1013 obtains threshold value process flow diagram

Fig. 6 is that in Fig. 1, square frame 1017 obtains parallel leak result process flow diagram

Fig. 7 is the data structure diagram that in Fig. 2, providing data formatting is used

Embodiment

It is as follows that the invention will be further described in conjunction with the accompanying drawings and embodiments.

Fig. 1 is process flow diagram of the present invention, and the embodiment that parallel leak is found is described.Described parallel leak is found to start to leak data acquisition and is carried out pre-service.Second step is for extracting leak set.The vectorization that the 3rd step is vulnerability information.The 4th step is for utilizing training set to obtain threshold value.The 5th step is for obtaining parallel leak result.

Fig. 2 is data acquisition and pretreatment process figure in Fig. 1, illustrates and how to obtain leak data and carry out a pretreated embodiment.Described flow process starts from from two public data sources, obtaining data (MITRE and CVSS), and next step carries out input system to raw data, consolidation form operation, form four data acquisition (CAPEC, CWE, CVE and CVSSscores), finally form the mapping that is identified to data.

Fig. 7 is data structure used in Fig. 2.

Fig. 3 extracts leak set process flow diagram in Fig. 1, first form CAPEC to the mapping (according to Related_Weakness) of CWE, then form CWE to the mapping (Observed_Example) of CVE, finally form CAPEC to the mapping of CVE, thereby huge leak set is split according to pattern.

Fig. 4 is the process flow diagram of Fig. 1 Chinese version vectorization, and the first step is obtained dictionary, and method, for traversal vulnerability database, is extracted the stem information that leak is described, and according to Rules Filtering, goes out antistop list.Second step, respectively by the description participle of each leak, extracts stem, and according to dictionary, removes non-key word, forms keyword sequence.The 3rd step is by the keyword sequence vectorization of leak, with TF/IDF plan value element vector.

Fig. 5 obtains the process flow diagram of threshold value in Fig. 1, first manually choose training set and, according to utilizing cosine similarity principle to calculate similarity, according to CVSS, can utilize mark compute euclidian distances, the threshold value of ascending the throne of averaging.

Fig. 6 obtains parallel leak result process flow diagram in Fig. 1, using the leak set obtaining in Fig. 3 as test set, order choose two different leaks, first calculate assessment fractional value, if be less than threshold value, continue, then calculate cosine similar value, if be greater than threshold value, be parallel leak.

According to the thinking of above-mentioned parallel bug excavation method and performing step, by the operation result of the parallel bug excavation prototype system based on increase income storehouse and text mining, know, finally can find that the parallel leak number of combinations obtaining is 2013 groups; According to practical experience, carry out finding after artificial verification have 192 groups not to be the leak that effectively walks abreast in the parallel leak combination that system obtains, accuracy is about 90.5%.

Aspect the performance test of prototype system, use the poor calculating corresponding speed of System.currentTimeMillis () method acquisition time herein, can obtain reaching several ten thousand in data stream, need to extract the lower system response time of situation of several ten thousand times.Such as, aspect calculated threshold, complete that to come from the response time that the CAPEC attack mode that comprises two and above relevant vulnerability amounts to the training set of 29 groups of data be 40ms; Aspect its parallel bug excavation, for processing response time of 62407 leaks in 400 attack modes be 35s.Incorporation engineering practical experience, native system possesses acceptable performance index.

Be in addition, analyze the Result example of prototype system, such as for attack mode CPAEC-157:Sniffing Attacks, by analyzing its attack step, associated weakness (CWE), precondition, postcondition, correlation technique and the applicable important informations such as technological accumulation and inheritance, according to said method step, the cosine similar value that can calculate leak is about 0.26, the value of the Euclidean distance of CVSS assessment mark is 0, combined training obtains the threshold value of cosine similar value of parallel leak and the threshold value of the Euclidean distance of CVSS assessment mark, analysis knows that these two all in threshold range, we can obtain its leak CVE-2009-1466 and CVE-2008-1567 is one group of parallel leak.

Known by above-mentioned experimental analysis, this paper method of utilization can be excavated corresponding parallel leak comparatively exactly, and has good system performance.

This paper Security-Oriented field, for information security leak association mining, study a question, based on storehouses of increasing income such as CAPEC, CWE and CVE, attempt introducing the thinking of parallel bug excavation herein, proposed a kind of parallel bug excavation method based on increase income storehouse and text mining.First be based on the information bank of increasing income, extract the associated fragility CWE under same attack mode, then be associated with leak CVE from fragility CWE, thereby be convenient to analyze concurrency relation potential between leak; Then by the text description information vector of leak, so that software systems are carried out intelligent processing method to leak record data; Be different from the inquiry based on keyword match, the threshold value drawing according to training set is investigated the similarity between leak, calculates the concurrency relation between leak, excavates parallel leak; Finally, verify by experiment and put into practice, having verified correctness and the validity of this paper method.The result obtaining according to this paper method can make up rapidly other parallel leak, and then make up the fragility of whole network when finding that a leak is utilized, and has higher cyber-defence using value.

Claims (1)

1. the parallel bug excavation method based on increase income storehouse and text mining, is characterized in that, step comprises:

1) data acquisition and pre-service:

A) from the storehouse of increasing income, obtain original attack mode information, vulnerability information and vulnerability information;

B) the library information data of increasing income of obtaining are carried out to data scrubbing, comprise Uniform data format, data purification, filling disappearance attribute and remove noise data;

C) data after cleaning are set up to data acquisition, and form the mapping that is identified to non-structured text information, form three set Pattern{p ₁, p ₂... p _n, Weak{w ₁, w ₂... w _m, Vulnerability{v1, v ₂... v ₃; P wherein _irepresent an attack mode, w _irepresent a fragility, v _irepresent a leak;

2) extract mapping relations and leak set

Parse attack mode p _ito fragility w _jmapping relations, fragility w _jto leak v _kmapping relations, then form attack mode p _ito leak v _kmapping relations, thereby huge vulnerability database is concluded to different leak set according to some attack modes;

3) the textual description information of leak is carried out to mathematical modeling:

D) extract the textual description information of leak, carry out participle, extract stem, according to feature dictionary, to describing, carry out text vector;

E) tf/idf that calculates document word according to the descriptor of text vector is that the word frequency/document that falls is worth frequently;

4) to step 2) in the leak of the leak set of extracting, according to the precondition of attack mode, postcondition and threshold value, screen, obtain the set of parallel leak combination;

A), according to the parallel leak training set of artificial mark, calculate the threshold value of cosine similar value of parallel leak and the threshold value of the Euclidean distance of CVSS assessment mark;

B) in step 2) in the leak set of obtaining, according to step 3) in the leak vector information that gets, the cosine similar value of calculating respectively two leaks corresponding to precondition and the postcondition of attack mode;

C) according to the Euclidean distance of the assessment mark of the cosine similar value of two leaks and CVSS, by two all the leak in threshold range separately differentiate for parallel leak.