CN103248609A

CN103248609A - System, device and method for detecting data from end to end

Info

Publication number: CN103248609A
Application number: CN2012100247861A
Authority: CN
Inventors: 孙睿; 李健航
Original assignee: Tongfang Co Ltd
Current assignee: Tsinghua Tongfang Co Ltd; Tongfang Co Ltd
Priority date: 2012-02-06
Filing date: 2012-02-06
Publication date: 2013-08-14

Abstract

The invention discloses a system, a device and a method for detecting data from end to end. The system comprises a first network card, a second network card, a data flow processing module, a data detection module and a threat strategy module. The device comprises a data flow processing module, a data detection module and a threat strategy module. The method comprises the steps as follows: data flowing into the first network card are scanned, so that necessary information required by detection is acquired; based on the necessary information, the sent data are segmented, known threat data are searched, whether data segments which are the same as the threat data exist is determined, and whether threat data exist in data per unit is judged according to a comparison result; compared data per unit and a corresponding judgment result are sent to the threat strategy module; and whether the compared data per unit is forwarded to the second network card is determined according to the judgment result sent by the data detection module. The system, the device and the method for detecting data from end to end, provided by the invention, can effectively improve the data detection accuracy.

Description

A kind of data detection system end to end, apparatus and method

Technical field

The present invention relates to the network security technology field, particularly relate to a kind of data detection system end to end, apparatus and method.

Background technology

The continuous development of the communication technology makes and the mutual of information and obtain more conveniently but also makes the faster propagation of unwelcome data such as some useless even harmful Viruses, spam and advertisement video simultaneously.For example the user is when downloading file, some lawless persons or businessman often are in commerce or other purposes, files such as Virus, advertisement hereof mix, this can bring adverse effect to normal operation of client, therefore how the data of transmission is effectively filtered and detection is an important topic.

Prevent that the computer of client is not subjected to the infringement of Virus etc., can come filter user not wish the program of seeing by fire compartment wall is set in client, also can be by in the data transmission procedure content of transmitting being detected end to end, Virus etc. is filtered out, make that the normal operation of client is unaffected.

The source end of end-to-end finger network and being connected of destination, network will be communicated by letter, and must connect, no matter space length is how far, middle through how many machines, all must between source end and destination, connect, set up in case connect, this kind connection just becomes connection end to end.

Based on content detection technique use end to end in the prior art is deep packet inspection technical, carry out the condition code comparison at packet header bag tail collection sample and viral fingerprint base, and then judge transmit data and whether comprise malicious code, this technology is widely used in UTM(Unified Threat Management, UTM at present) and PAA in the middle of.

This kind deep packet inspection technical only detects packet header and bag portion, is a kind of sampling techniques of thick leakage, and processing speed is fast, and computational requirements is smaller, is easy to handle small documents.Though this is a kind of more cost effective mode, can reduce the computing cost, but may be flooded with spam, advertisement video and institute of enterprise inappreciative P2P transmission supervisor in the data load of file, and the HTML of various ecommerce programs (Hypertext Markup Language, HTML) and XML(Extensible Markup Language, extend markup language) also secretly the back door may carried in the formatted data and trojan horse program exchanges between network node.So, in today that application form and form thereof increase with detonation velocity, only the packet header of data-driven bag and bag tail determine its whether detection mode of this thick leakage of access, can make unwelcome program produce serious harm by the defence border in security system inside, can't satisfy safe requirement, i.e. prior art based on the content detection accuracy is not high end to end.

Summary of the invention

The object of the present invention is to provide a kind of data detection system end to end, apparatus and method, it has solved the not high problem of deep packet inspection technical accuracy of the prior art.

A kind of data detection system end to end for realizing that the object of the invention provides comprises first network interface card and second network interface card, also comprises data traffic processing module, data detection module, threat strategy module; Described data traffic processing module, the data that are used for the inflow of scanning first network interface card detect required necessary information to obtain; Described data detection module, be used for according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module transmission, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module through the unit data of contrast, corresponding judged result; Described threat strategy module is used for determining whether described unit data through contrast is forwarded to described second network interface card according to the judged result that described data detection module sends.

Wherein, described data detection module comprises segmentation module, threatens fingerprint base and threatens judge module; Described segmentation module is used for the data after the described data traffic processing module scanning are carried out segmentation and be transferred to threatening judge module 33; Described threat fingerprint base is the database of the known threat data of a storage; Described threat judge module, be used for searching for according to the segment data after the segmentation module segmentation threat data of described threat fingerprint base, confirm whether to have identical with it data segment, will with described threat fingerprint base in the data segment that mates fully of described threat data be judged as threat data, to be judged as secure data with the data segment that the threat data that threatens in the fingerprint base not exclusively mates, and will be sent to described threat strategy module through unit data and the corresponding judged result of contrast.

Wherein, also comprise the cache optimization module; Be used for each file is generated a file record, described file record comprises file SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in the current file record according to the history file record, and the current file record is sent to described threat strategy module; Described historical fingerprint buffer memory storehouse is used for storage and upgrades all through the file record of the file of detection.

Wherein, described data traffic processing module, also be used for setting up main line task management tabulation, be responsible for the subtask management of data detection module and cache optimization module, and main line task management tabulation is sent to the threat strategy module, again the main thread mission number is sent to data detection module and cache optimization module.

Wherein, described cache optimization module also is used for according to the unit data that belongs to complete file that receives, make up sub-line mission number, and module numbering, main line mission number, sub-line mission number and task status task management data are sent to described threat strategy module.

Wherein, described data detection module also is used for the main line mission number according to the data traffic processing module, make up sub-line mission number and sub-line task module numbering, after handling total data unit, each complete file according to the SHA-1 algorithm, is produced the signature of each file, described signature comprises settling time, the SHA-1 fingerprint value, main line mission number, sub-line task module numbering, sub-line mission number and task status.

Wherein, described cache optimization module comprises generation logging modle, historical fingerprint buffer memory storehouse and threatens historical judge module; Described generation logging modle, be used for according to the unit data that belongs to complete file that receives, make up sub-line mission number, and sub-line task module numbering, main line mission number, sub-line mission number and task status be sent to described threat strategy module, also be used for the spanned file record, described file record comprise file the SHA-1 field, whether hit the buffer memory field, whether detected field, whether belong to and threaten field and whether carry out the clearance operation field; The historical judge module of described threat is used for whether having the history file that has identical SHA-1 fingerprint value with current file in the middle of the described historical fingerprint buffer memory of the retrieval storehouse; If there is the history file that has identical SHA-1 fingerprint value with current file, then the value of whether hitting the buffer memory field of described current file is written as and is; If threaten field for empty whether belonging to during described history file records, then whether the detected field in the described current file record being written as is to deny otherwise write; With the value that threatens field that whether belongs to that the value that threatens field writes described current file that whether belongs in the described history file record; The described clearance operational word segment value of whether carrying out is for empty; Do not have the history file that has identical SHA-1 fingerprint value with current file, then the buffer memory field of whether hitting with described current file is written as not, whether described detected field is write not, the described threat field that whether belongs to is write sky, whether carries out the value of clearance operation field for empty; The current file record is back to described threat strategy module.

Wherein, described threat strategy module also is used for determining whether described data through contrast are forwarded to described second network interface card strategy that whether allows threat data to let pass that described threat strategy sets up on their own for the user according to threat strategy.

Wherein, described threat strategy module comprises threat strategy unit, blocking unit and data forwarding register cell; Described threat strategy unit, be used for the main line task management tabulation that receiving data stream amount processing module is imported into, storage main line task management tabulation, the signature of the file that the judge module of reception threat simultaneously imports into and the task management data that threaten historical judge module to import into, the file record, set up index with the main line mission number in the signature of file, insert sub-line mission number and sub-line task module numbering, insert the task completion status, and the judged result and the described threat strategy that return according to described threat judge module, carry out forwarding or the blocking-up of data: when judged result when belonging to secure data, then the data of correspondence are let pass, be sent to described data and transmit register cell; The judged result of returning when described threat judge module then sends a reminder message to the user and does not carry out the threat data that has obtained to remind the user for belonging to threat data and threat strategy when letting pass when transmitting data; When the judged result of returning when described threat judge module was set to block for belonging to threat data and threat strategy, then the data with correspondence were sent to blocking unit; Described data are transmitted register cell, are used for being forwarded to second network interface card by the data that described threat strategy unit sends; Described blocking unit is used for and will be abandoned by the threat data that described threat strategy unit sends, and ends the detection to described threat data place file further part, and the data that stop to detect will all abandon.

Wherein, described threat strategy unit, also be used for the described data detection module of management and described cache optimization module: when described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for being, when whether belonging to threat field non-NULL, being that index finishes described data detection module to the detection of current file record institute respective file with the main thread task ID in this document record; When described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for not the time, after waiting for that then described data detection module finishes the detection of current file record respective file, whether belong to the threat field with what the judged result of returning write the current file record; When described data detection module is returned judged result to described threat strategy module prior to described cache optimization module, then finish described cache optimization module about the execution in step of described judged result respective file.

Wherein, described data are transmitted register cell, also be used for when the file record of current file does not have the SHA-1 value, then calculate the SHA-1 fingerprint value of current file, and the SHA-1 fingerprint value of calculating and the described judged result that whether belongs to threat data of described threat judge module transmission are sent to described historical fingerprint buffer memory storehouse.In the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert simultaneously, file SHA-1 value, whether belong to threats (be/not), system's detection time (Year/Month/Day), the fingerprint classification that detects (virus, spam, network address are classified, code injection), detection fingerprint base version number, other descriptions (sky).

Wherein, described generation logging modle also comprises the P2P protocol processing unit; When described P2P protocol processing unit, the SHA-1 fingerprint value that in the cache optimization module P2P agreement is provided are carried out the judgement of history threat fingerprint, can select three options for use, fully trust, trust authentication, distrust fully; In the cache optimization module, when selecting to trust fully, the SHA-1 fingerprint value that provides in the use P2P agreement is not detained data as the SHA-1 signature of file; When selecting trust authentication, the SHA-1 fingerprint value that provides in the P2P agreement then is provided, and use this SHA-1 fingerprint value in threatening historical judge module, to judge, after the judged result generation, also require data detection module after having loaded file data, last packet of suppressing a document is waited for because the SHA-1 fingerprint value generation of spanned file finishes.If the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is identical, then last packet is let pass, if the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is inequality, then use newly-generated file SHA-1 fingerprint value threatening historical judge module to carry out fingerprint threat judgement.When selecting to distrust fully, then make the SHA-1 fingerprint value of spanned file carry out the judgement of history threat fingerprint.

The present invention also provides a kind of data detection device end to end, is connected end to end between first network interface card and second network interface card, and, comprise data traffic processing module, data detection module, threat strategy module and cache optimization module; Described data traffic processing module, the data that are used for the inflow of scanning first network interface card detect required necessary information to obtain; Described data detection module, be used for according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module transmission, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module through the unit data of contrast, corresponding judged result; Described threat strategy module is used for the judged result according to described data detection module transmission, sets with reference to threat strategy to determine whether described unit data through contrast is forwarded to described second network interface card; Described cache optimization module is used for each file is generated a file record, and described file record comprises file SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in generation current file record according to the history file record, and the current file record is sent to described threat strategy module; Described historical fingerprint buffer memory storehouse is used for storage and upgrades all through the historical information record of the file of detection.

The present invention also provides a kind of method of Data Detection end to end, comprises step: steps A. and the data that described data traffic processing module scans the inflow of first network interface card detect required necessary information to obtain; The described data detection module of step B. is according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module transmission, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module through the unit data of contrast, corresponding judged result; The judged result that the described threat strategy module of step C. sends according to described data detection module determines whether described unit data through contrast is forwarded to described second network interface card.

Wherein, among the described step B, describedly judge whether there is threat data in the described unit data according to comparing result, comprise the steps: with described threat fingerprint base in the data segment that mates fully of described threat data be judged as threat data; Will with described threat fingerprint base in threat data not exclusively the data segment of coupling be judged as secure data.

Wherein, described step C comprises the steps: that also described threat strategy module also determines whether described unit data through contrast is forwarded to described second network interface card according to threat strategy.

Wherein, described step C comprises: the judged result that step C1. returns when described data detection module is when belonging to secure data, then the unit data with correspondence is forwarded to second network interface card, the judged result of returning when described data detection module is for belonging to threat data and threat strategy when letting pass, then the unit data with correspondence is forwarded to second network interface card, sends reminder message to the user simultaneously and does not open the file at threat data place to remind the user; When the judged result that step C2. returns when described data detection module was set to block for belonging to threat data and threat strategy, then the data with correspondence abandoned, and ended the detection to the further part of threat data place file.

Wherein, described step B also comprises: step B '. the cache optimization module generates a file record to each file, and described file record comprises file SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in generation current file record according to the history file record, and the current file record is sent to described threat strategy module.

Wherein, described steps A also comprises step: set up main line task management tabulation, be responsible for the subtask management of data detection module and cache optimization module, and main line task management tabulation is sent to the threat strategy module, again the main thread mission number is sent to data detection module and cache optimization module.

Wherein, described step B also comprises step: according to the main line mission number of data traffic processing module, make up sub-line mission number and sub-line task module numbering, after handling total data unit, each complete file according to the SHA-1 algorithm, is produced the signature of each file.

Wherein, described step B ' also comprises step: according to the unit data that belongs to complete file that receives, make up sub-line mission number, and module numbering, main line mission number, sub-line mission number and task status task management data are sent to described threat strategy module.

Wherein, described step B ' comprising: step B1 '. according to the unit data that belongs to complete file that receives, make up sub-line mission number, and sub-line task module numbering, main line mission number, sub-line mission number and task status are sent to described threat strategy module, spanned file record; Step B2 '. whether the historical judge module retrieves historical of described threat fingerprint buffer memory exists the history file that has identical SHA-1 fingerprint value with current file in the middle of the storehouse; There is the history file that has identical SHA-1 fingerprint value with current file, then the value of whether hitting the buffer memory field of described current file is written as and is; If threaten field for empty whether belonging to during described history file records, then whether the detected field in the described current file record being written as is to deny otherwise write; With the value that threatens field that whether belongs to that the value that threatens field writes described current file that whether belongs in the described history file record; The described clearance operational word segment value of whether carrying out is for empty; Do not have the history file that has identical SHA-1 fingerprint value with current file, then whether the buffer memory field of whether hitting with described current file is not written as not, will described detected field writes not, the described threat field that whether belongs to is write sky, whether carries out the clearance operation field and write sky; Step B3 '. the historical judge module of described threat is back to described threat strategy module with the current file record.

Wherein, after the described step B, also comprise before the described step C: step C1 '. when described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for being, when whether belonging to threat field non-NULL, being that index finishes described data detection module to the detection of current file record institute respective file with the main thread task ID in this document record; When described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for not the time, after waiting for that then described data detection module finishes the detection of current file record respective file, whether what the judged result that described threat strategy unit is returned write the current file record belongs to the threat field; Step C2 '. when described data detection module was returned judged result to described threat strategy module prior to described cache optimization module, then described threat strategy unit finished described cache optimization module about the execution in step of described judged result respective file.

Wherein, also comprise after the described step C: when described data forwarding register cell does not have the SHA-1 value in the file record of current file, then calculate the SHA-1 fingerprint value of current file, and the SHA-1 fingerprint value of calculating and the described judged result that whether belongs to threat data of described threat judge module transmission are sent to described historical fingerprint buffer memory storehouse; In the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert simultaneously file SHA-1 fingerprint value, fingerprint classification (virus, spam, network address classification, code injection), detection fingerprint base version number, other descriptions (sky) of whether belonging to threats (be/not), system's detection time (Year/Month/Day), detecting.

Wherein, when the file of first network interface card inflow is the P2P document of agreement, then after described steps A, step B1 ' before, also comprising: steps A 1 '. the SHA-1 fingerprint value that the cache optimization module provides the P2P agreement carries out history and threatens fingerprint to judge, be provided with three options, fully trust, trust authentication, distrust fully; Steps A 2 '. in the cache optimization module, when selecting to trust fully, the SHA-1 fingerprint value that provides in the use P2P agreement is not detained data as the SHA-1 signature of file; When selecting trust authentication, the SHA-1 fingerprint value that provides in the P2P agreement then is provided, and use this SHA-1 fingerprint value in threatening historical judge module, to judge, after the judged result generation, also require data detection module after having loaded file data, last packet of suppressing a document waits the SHA-1 fingerprint value that is ready to use in spanned file to generate and finishes.If the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is identical, then last packet is let pass, if the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is inequality, then use newly-generated file SHA-1 fingerprint value threatening historical judge module to carry out fingerprint threat judgement; When selecting to distrust fully, then make the SHA-1 fingerprint value of spanned file carry out the judgement of history threat fingerprint.

The invention has the beneficial effects as follows: a kind of data detection system end to end provided by the invention, apparatus and method, the method that compares with known threat data is carried out after the segmentation by turn in employing to the data that transmit between end-to-end, judge in the current detected file and whether have threat data, because the total data content of file has all been passed through detection, therefore accuracy sampling Detection accuracy compared to prior art increases, and the problem of threat data can not occur missing; In addition, the detection method of prior art detect contrast to as if whole file, and the detection of technical scheme provided by the present invention contrast to as if segmentation after segment data, the shared more whole file of memory headroom of segment data is littler; Therefore a kind of data detection system end to end provided by the present invention, apparatus and method, when improving accuracy in detection, minimizing equipment calculates and storage overhead.

Description of drawings

Fig. 1 is the structural representation of a kind of execution mode of a kind of data detection system end to end of the present invention;

Fig. 2 is the structural representation of the another kind of execution mode of a kind of data detection system end to end of the present invention;

Fig. 3 is the flow chart of a kind of execution mode of a kind of method of Data Detection end to end of the present invention;

Fig. 4 is the flow chart of the another kind of execution mode of a kind of method of Data Detection end to end of the present invention.

Embodiment

In order to make purpose of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, a kind of data detection system end to end of the present invention, apparatus and method are further elaborated.Should be appreciated that specific embodiment described herein only in order to explaining the present invention, and be not used in restriction the present invention.

Embodiment one

The invention provides a kind of data detection system end to end, be used for the data in transmission between the end of network and the end are contrasted detection one by one, with files such as wherein Virus of filtration, spam, advertisement videos.As shown in Figure 1, as a kind of embodiment, this system comprises: first network interface card 1, data traffic processing module 2, data detection module 3, threat strategy module 4 and second network interface card 5.

The data that described data traffic processing module 2 is used for 1 inflow of scanning first network interface card detect required necessary information to obtain.Described necessary information comprises forms the sequence of data packet set that enters in the detection unit data, affiliated agreement and communication source and destination address, and wherein unit data is the partial data of unit for the character string with file or agreement length.And with necessary information and transfer of data to data detection module 3.

Described data detection module 3, be used for according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module 2 transmissions, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module 4 through the unit data of contrast, corresponding judged result.

Described data detection module 3 comprises segmentation module 31, threatens fingerprint base 32 and threatens judge module 33.

Described segmentation module 31 is used for the data after the described data traffic processing module scanning are carried out segmentation and be transferred to threatening judge module 33.

Described threat fingerprint base 32 is the database of the known threat data of a storage.

Described threat judge module 33 is used for searching for according to the segment data after segmentation module 31 segmentations threat data of described threat fingerprint base 32, confirm whether to have identical with it data segment, will with described threat fingerprint base 32 in the data segment that mates fully of described threat data be judged as threat data, to be judged as secure data with the data segment that the threat data that threatens in the fingerprint base 32 not exclusively mates, and will be sent to described threat strategy module 4 through unit data and the corresponding judged result of contrast.

Described threat strategy module 4 is used for the judged result according to described data detection module 3 transmissions, sets with reference to threat strategy to determine whether described unit data through contrast is forwarded to described second network interface card 5.When the judged result that receives from described data detection module 3 is clearance for belonging to threat data and threat strategy, at first unit data is forwarded to second network interface card 5 and sends a reminder message to the user and do not carry out the threat data that has obtained to remind the user; When the judged result that receives from the threat judge module of described data detection module 3 was set to block for belonging to threat data and threat strategy, then the unit data data with correspondence were sent to blocking unit.

As shown in Figure 2, as a kind of more excellent execution mode, a kind of data detection system end to end of the present invention comprises first network interface card 1 and second network interface card 5, data traffic processing module 2, data detection module 3, threat strategy module 4 and cache optimization module 6.

The data that described data traffic processing module 2 is used for 1 inflow of scanning first network interface card detect required necessary information to obtain.

Described necessary information comprises forms the sequence of data packet set that enters in the detection unit data, affiliated agreement and communication source and destination address, and wherein unit data is the partial data of unit for the character string with file or agreement length.

To data detection module 3, simultaneously, the unit data that described data traffic processing module 2 only will belong to complete file is sent to cache optimization module 6 to described data traffic processing module 2 with necessary information and transfer of data.

Described data traffic processing module 2 also is used for setting up main line task management tabulation, is responsible for the subtask management of data detection module 3 and cache optimization module 6, and main line task management tabulation is sent to threat strategy module 4.Again the main thread mission number is sent to data detection module 3 and cache optimization module 6.Main line task management tabulation comprises time, main task numbering, data detection module 3 and cache optimization module 6 subtask module numberings, the task status that described partial data is set up.

The main line mission number that described data detection module 3 is used for according to data traffic processing module 2, make up sub-line mission number, and module numbering, main line mission number and sub-line mission number be sent to described threat strategy module 4, simultaneously according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module 2 transmissions, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module 4 through the data of contrast, corresponding judged result.

The same a kind of execution mode, described data detection module 3 comprises segmentation module 31, threatens fingerprint base 32 and threatens judge module 33 in the present embodiment.

Described threat judge module 33 in the described data detection module 3, except the function described in a kind of execution mode before having, also be used for after handling total data unit, each complete file according to the SHA-1 algorithm, is carried out the calculating of the integrality signature of data, and the signature that each file calculating is produced is delivered to threat strategy module 4, described signature comprises settling time, the SHA-1 value, main line mission number, sub-line task module numbering, mission number and task status.The signature of each file is unique.

Described cache optimization module 6 is used for according to the unit data that belongs to complete file that receives, make up sub-line mission number, and task management data such as module numbering, main line mission number, sub-line mission number and task status are sent to described threat strategy module 4.

Described cache optimization module 6 also is used for each file is generated a file record, and described file record comprises file integrality SHA-1 (SHA is a kind of security algorithm that does not have the logic conflict that use is recommended by safety of America office) fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in generation current file record according to the history file record, and the current file record is sent to described threat strategy module 4.

Described cache optimization module 6 comprises generation logging modle 61, historical fingerprint buffer memory storehouse 62 modules and threatens historical judge module 63.

Described generation logging modle 61, be used for according to the unit data that belongs to complete file that receives, make up sub-line mission number, and task management data such as module numbering, main line mission number, sub-line mission number and task status are sent to described threat strategy module 4, be used for the spanned file record simultaneously, described record comprise file the SHA-1 field, whether hit the buffer memory field, whether detected field, whether belong to and threaten field and whether carry out the clearance operation field.

The historical judge module 63 of described threat, be used for whether having the history file that has identical SHA-1 field value with current file in the middle of the retrieves historical fingerprint buffer memory storehouse, fill in the current file record according to the history file record, and the current file record is sent to described threat strategy module 4.

Store the historical information record of before scanned All Files in the described historical fingerprint buffer memory storehouse 62.

Described generation logging modle 61 also comprises the P2P protocol processing unit; Described P2P protocol processing unit is used for can selecting three options for use, fully trust, trust authentication, distrust fully when the SHA-1 fingerprint value that the cache optimization module provides the P2P agreement carries out the judgement of history threat fingerprint; In the cache optimization module, when selecting to trust fully, the SHA-1 fingerprint value that provides in the use P2P agreement is not detained data as the SHA-1 signature of file; When selecting trust authentication, the SHA-1 fingerprint value that provides in the P2P agreement then is provided, and use this SHA-1 fingerprint value in threatening historical judge module, to judge, after the judged result generation, also require data detection module after having loaded file data, last packet of suppressing a document waits the SHA-1 fingerprint value that is ready to use in spanned file to generate and finishes.If the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is identical, then last packet is let pass, if the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is inequality, then uses newly-generated file SHA-1 fingerprint value to carry out history and threaten fingerprint to judge.

In the present embodiment, described threat strategy module 4 comprises threat strategy unit 41, blocking unit 43 and data forwarding register cell 42.

Described threat strategy unit 41 is used for the main line task management tabulation that receiving data stream amount processing module 2 is imported into, storage main line task management tabulation, the signature of the file that the judge module 33 of reception threat simultaneously imports into and task management data, the file record that threatens historical judge module 63 to import into, set up index with the main line mission number in the signature of file, insert sub-line mission number and sub-line task module numbering, insert the task completion status.

Forwarding or the blocking-up of data are carried out also for the judged result and the described threat strategy that return according to described threat judge module 33 in described threat strategy unit 41.

When judged result when belonging to secure data, then the data of correspondence are let pass, be sent to described data and transmit register cell 42, will be sent to data by the signature of the place file of clearance data, file record and task management data simultaneously and transmit register cell 42; The judged result of returning when described threat judge module 33 then sends a reminder message to the user and does not carry out the threat data that has obtained to remind the user for belonging to threat data and threat strategy when letting pass when transmitting data; When the judged result of returning when described threat judge module 33 was set to block for belonging to threat data and threat strategy, then the unit data with correspondence was sent to blocking unit 43.

Described data are transmitted register cell 42, are used for being forwarded to second network interface card 5 by the data that described threat strategy unit 41 sends.

Described data are transmitted register cell 42 and also are used for when the file record of current file does not have the SHA-1 value, then calculate the SHA-1 fingerprint value of current file, and the described judged result that whether belongs to threat data that the SHA-1 fingerprint value that calculates and described threat judge module 33 return is sent to described historical fingerprint buffer memory storehouse 62.In the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert simultaneously, file SHA-1 value, whether belong to threats (be/not), system's detection time (Year/Month/Day), the fingerprint classification that detects (virus, spam, network address are classified, code injection), detection fingerprint base version number, other descriptions (sky).

Described blocking unit 43 is used for and will be abandoned by the threat data that described threat strategy unit 41 sends, and ends the detection to described threat data place file further part, and the data that stop to detect will all abandon.

41 described threat strategy unit, described threat strategy unit also are used for the described data detection module 3 of management and described cache optimization module 6.

When described cache optimization module is recorded to described threat strategy module 4 prior to described data detection module 3 backspace files, and whether the value of the described detected field in the current file that described cache optimization module 6 the is returned record for being, when whether belonging to threat field non-NULL, being the detection that index finishes 3 pairs of current files records of described data detection module institute respective file with the main thread task ID in this document record.

When described cache optimization module 6 is recorded to described threat strategy module 4 prior to described data detection module 3 backspace files, and whether the value of the described detected field in the current file that described cache optimization module 6 the is returned record for not the time, after the detection of then waiting for the 3 pairs of current files of described data detection module record respective file finishes, whether belong to the threat field with what the judged result of returning write the current file record.

When described data detection module 3 is returned judged result to described threat strategy module 4 prior to described cache optimization module 6, then finish described cache optimization module 6 about the execution in step of described judged result respective file.

The present invention also provides a kind of data detection device end to end, be connected end to end between first network interface card 1 and second network interface card 5, as a kind of embodiment, comprise data traffic processing module 2, data detection module 3, threat strategy module 4 and cache optimization module 6.

Described data traffic processing module 2, the data that are used for 1 inflow of scanning first network interface card detect required necessary information to obtain.

Described threat strategy module 4 is used for determining whether described unit data through contrast is forwarded to described second network interface card 5 according to the judged result that described data detection module 3 sends.

A kind of data detection system end to end provided by the present invention and device, by being set, 3 pairs of data that flow into through an end of network of data detection module carry out careful detection, and cache optimization module 6 is set retrieves whether current file is before tested, after definite current file was before tested, the detection of end process, thus the efficient that detects when improving accuracy in detection, improved effectively.

Embodiment two

The present invention also provides a kind of method of Data Detection end to end, and as a kind of embodiment, as shown in Figure 3, a kind of method of Data Detection end to end provided by the invention comprises step:

Step S100, described data traffic processing module scans the data that first network interface card 1 flows into and detects required necessary information to obtain, and with necessary information and transfer of data to data detection module 3.

The data that are in transport layer flow into by first network interface card 1, mode with Transparent Proxy is passed to described data traffic processing module, when receiving the data that flow into via first network interface card 1, the method of Data Detection end to end of the embodiment of the invention is at first resolved HTTP (the HyperText Transfer Protocol that recognition data is observed in transmission course, HTML (Hypertext Markup Language)), FTP (File Transfer Protocol, file transfer protocol (FTP)), SMTP(Simple Mail Transfer Protocol, Simple Mail Transfer protocol), POP3(Post Office Protocol, Post Office Protocol 3 originally), IMAP(Internet Mail Access Protocol, interactive email access agreement) agreement such as.Preferably, in the embodiment of the invention, further resolve identification SSL (Secure Sockets Layer, SSL), P2P (Peer to Peer, point-to-point), VPN(Virtual Private Network, VPN (virtual private network)) etc. various protocols.

Behind the file destination when determined the data that flow into through first network interface card by scanning in, making up one is the task execution tabulation of unit with the file, and essential information is write this tabulation.Determining concrete file destination by scanning, is that unit makes up follow-up detection task with each concrete file.As a kind of embodiment, can be the corresponding testing process of a file, also can a plurality of files be detected a testing process simultaneously.

As a kind of embodiment, to carry out in the tabulation in task, the corresponding data of each file detect task and task management task, and the task of described Data Detection namely is the concrete testing process to file, is finished by described data detection module 3; Described task management task then is that whole technical proposal is dispatched and managed, and is finished by described threat strategy module 4.The purpose that the structure task is carried out tabulation is to call relevant task program.

Described data traffic processing module 2 performed processes can be made as parent process as a kind of embodiment, and data detection module 3 performed steps are made as the first sub-thread, described threat strategy module 4 performed steps are made as the sub-thread of task management, coordinate and manage a plurality of tasks by the mode of parent process and sub-thread like this.

Work out the task ID (identity of described parent process, numbering), the task ID of the described first sub-thread and the mission number of the sub-thread of described task management, and each task executions state is set, as each task of a kind of embodiment three kinds of executing states are arranged: carrying out, carrying out end, parallel task.Then can realize the information transmission between each process and call by mission number and task status, be easier to the management of a plurality of processes.

As a kind of embodiment, comprised the numbering of data agreement (http, ftp, pop3, smtp etc.), fileinfo (all packet numbers of loading or part stress state, filename, combination), each tasks when transmission etc. in the described listed files.

Described step S100 is the preparation of carrying out subsequent step, and the necessary information that file is scanned is provided.

Step S110. with the known threat data of search after the data segmentation of described data traffic processing module transmission, confirms whether identical with it data segment is arranged according to described necessary information, judges whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module through the unit data of contrast, corresponding judged result.

The process of described step S110 for the data content of file is detected in detail contrasts one by one to the concrete data content of file, in case in the middle of data threat data under cover, specifically comprise the steps:

Data after step S111. scans described data traffic processing module carry out segmentation and are transferred to threatening judge module 33.

Be the packet of file to be cut apart with regular length and skew goes to contrast by turn with threat data to the segmentation of file, relevant with the computing capability of equipment, handling property is more high, then simultaneously treated number of fragments is more many, can consider to increase simultaneously treated number of fragments, also can adjust the length of segmentation, mate relevant computing capability.Specifically cutting apart length and deflected length can be determined according to actual conditions by those skilled in the art, does not do restriction herein.

Step S112. searches for threat data in the described threat fingerprint base with the segment data after the segmentation module segmentation, confirms whether to have identical with it data segment.

In the comparison process, begin to compare with threatening the threat data in the fingerprint base with first position of cutting apart length, if primary importance is found coupling then generated matched record, coupling backward one by one again.For example, during the primary importance coupling, find 1,000 doubtful, be to have the primary importance of 1,000 files identical with the primary importance of described file fragmentation data in the middle of the described threat fingerprint base, carry out the coupling of the second place then, at this moment, the second place that may have only 100 threat datas is also identical with second of described file fragmentation data, by that analogy, contrasted up to the total data with described scanned document.

Described threat fingerprint base is the database of all known threat datas of storage, described threat data is all known disadvantageous paper samples of normal operation to custom system, include but not limited to all known viruses, spam, advertisement video, the purpose of the testing process that described data detection module is performed namely is to detect in the file that flows into through first network interface card whether be concealed with threat data.As a kind of embodiment, described threat fingerprint base can comprise that virus base, spam storehouse, network address are filtered the storehouse and the storehouse is injected in the website.

In the data transmission procedure of reality, the entire packet loaded of file can appear or only the portion of loading in two kinds of situation, if the packet of current file is whole loadeds, then can be with the order of file content, data content and the threat data contrast of length will fixedly be cut apart in the packet, can directly abandon after each segment data comparison finishes, at this moment workload is smaller.

If when contrast file whole loadeds then can compare each segment data by the sequencing that packet enters, and whether whether it is complete to can take into account in per two continuous packets similarity, need to be offset and comparison again.Such as, in the 3rd packet, scanning that the 10th byte begin may be similar to threat data, but comparison does not finish just do not had packet during the 3rd data end-of-packet, the data that at this moment can keep the 3rd packet, wait for the data of the 4th packet, up to the data arrives of the 4th packet and finish when detecting and do not have threat data, just being judged as does not have threat data.

If the 3rd packet then, the 2nd packet also do not arrive, though can detect the 3rd packet, will consider also whether the data of the 2nd packet and the 1st packet are also no problem.So the 3rd packet can be detained, all detected not have problems up to the data of the 1st packet and the 2nd packet and just can judge there is not threat data.

If when judging the 3rd packet, there is not threat data, and there is the threat data possibility the 2nd packet the inside, then can couple together with the data content of the 3rd packet, threat data starting position with the 2nd packet adds that the ED position of the 2nd packet begins, connect the 3rd packet, continue the Data Detection of the 2nd packet+3rd packet.The 2nd packet and the 3rd packet are let pass if the data content joint-detection of the 2nd packet+3rd packet is no problem.If it may be that problem is arranged that the joint-detection of the 2nd packet+3rd packet also is considered to, then unite the 4th packet again and carry out the joint-detection comparison.

Step S113. will with described threat fingerprint base in the data segment that mates fully of described threat data be judged as threat data, will with described threat fingerprint base in threat data not exclusively the data segment of coupling be judged as secure data.

Judge whether to belong to the threat data category according to matching degree, when in the file that is scanned and all threat datas that threaten in the middle of the fingerprint base any one mated fully, then assert to belong to threat data, otherwise do not belong to threat data not in full conformity with regarding as.

Step S114. will be sent to described threat strategy module through unit data and the corresponding judged result of contrast.

All do not finish the file that detects and do not load fully, all can detain last bag, and stage is abandoned intact after testing packet, thus the release device memory headroom.After the whole judgements of data threat are clear, last bag of then can letting pass.The partial data of file can be sent to described threat strategy module, further be sent to the user by parent process again, send again after all Data Detection that needn't wait until file are finished, reduce the time of delay of user's pending file data like this.

In actual mechanical process, as a kind of embodiment, can be index with the mission number, carry out between a plurality of tasks coordination with communicate by letter.Therefore, return to the mission number that also comprises the first sub-thread of described threat strategy module 4 simultaneously.

The judged result that step S120. threat strategy module 4 sends according to described data detection module 3 is set the unit data that determines whether described process contrast with reference to threat strategy and is forwarded to described second network interface card 5.

Whether described judged result exists threat data in the middle of showing current file; Described threat strategy is the strategy that whether allows threat data to let pass that the user sets up on their own, and whether the reaction user takes the measure blocked at threat data.

As a kind of embodiment, in the embodiment of the invention, when data are let pass or blocked, described judged result and described threat strategy are taken into consideration:

That is, the judged result of returning when described data detection module 3 is when belonging to secure data, and then the threat strategy no matter user arranges is all let pass the data of correspondence for letting pass or blocking-up, is forwarded to second network interface card; The judged result of returning when described data detection module 3 then sends a reminder message to the user and does not carry out the threat data that has obtained to remind the user for belonging to threat data and threat strategy when letting pass when transmitting data; When the judged result of returning when described threat judge module is set to block for belonging to threat data and threat strategy, then the data with correspondence abandon, and the testing process of ending the further part of threat data place file, directly the further part with this threat data place file abandons.

A kind of method of Data Detection end to end provided by the present invention, to contrasting one by one with known threat data after the data segmentation of file, prevent from missing threat data, thereby accuracy is higher, secondly, owing to the mechanism that adopts file fragmentation contrast, real time scan, real-time unit of transfer data, the memory headroom that makes this data detection process of execution take is little, the transmission data are fast, has reduced user's stand-by period when improving accuracy.

Embodiment three

As shown in Figure 4, as a kind of more excellent execution mode, a kind of method of Data Detection end to end provided by the present invention comprises step:

Step S200. scans the data that first network interface card flows into and carries out the required necessary information of subsequent step to obtain, and sets up main line task management tabulation, and relevant information is sent to data detection module 3, threat strategy module 4 and cache optimization module 6 respectively.

The data that described data traffic processing module 2 scannings first network interface card 1 flows into detect required necessary information to obtain, with necessary information and transfer of data to data detection module 3, simultaneously, described data traffic processing module 2 unit data that only will belong to complete file is sent to cache optimization module 6.

In the present embodiment, described data traffic processing module 2, also can carry out following steps: set up main line task management tabulation, be responsible for the subtask management of data detection module 3 and cache optimization module 6, and main line task management tabulation is sent to threat strategy module 4.Again the main thread mission number is sent to data detection module 3 and cache optimization module 6.Main line task management tabulation comprises time, main task numbering, data detection module 3 and cache optimization module 6 subtask module numberings, the task status that described partial data is set up.

The described data detection module of step S210. is handled the inter-related task management data, and data are carried out segmentation contrast back transmission judged result and file signature.

According to the main line mission number of data traffic processing module, make up sub-line mission number, and module numbering, main line mission number and sub-line mission number are sent to described threat strategy module 4.

Described threat judge module 33 in the described data detection module 3, after handling total data unit, to each complete file according to the SHA-1 algorithm, carry out the calculating of the integrality signature of data, and each file is calculated the signature that produces be delivered to threat strategy module 4, described signature comprises settling time, the SHA-1 value, the main line mission number, sub-line task module numbering, mission number and task status.

According to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module 2 transmissions, confirm whether identical with it data segment is arranged simultaneously, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module 4 through the data of contrast, corresponding judged result.

Wherein, segmentation is dynamic, is by file being cut apart with regular length and be offset, and each file data segmentation is contrasted with threat data by turn.

At first, begin to compare with threatening fingerprint base with first position of cutting apart length, if first is found coupling then generate matched record, coupling backward one by one contrast matched record afterwards and can characterize matching degree again.Matching degree meets fully and belongs to threat data, not in full conformity with not belonging to threat data.After whether having threat data in the file of determining to be scanned, this judged result is returned to described threat strategy module 4.

Step S220. cache optimization module is according to the unit data that belongs to complete file that receives, make up sub-line mission number, and task management data such as module numbering, main line mission number, sub-line mission number and task status are sent to described threat strategy module 4, each file is generated a file record, and described file record comprises file integrality SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in the current file record according to the history file record, and the current file record is sent to described threat strategy module 4.

In the present embodiment, step S220 is parallel to step S210, and the concrete implementation of described B step S220 is as follows:

According to the unit data that belongs to complete file that receives, make up sub-line mission number, and task management data such as module numbering, main line mission number, sub-line mission number and task status are sent to described threat strategy module 4.

Spanned file record, described file record comprise file the SHA-1 field, whether hit the buffer memory field, whether detected field, whether belong to and threaten field and whether carry out the clearance operation field.

The current file that needs retrieval of buffer memory, after the data of file are by complete storage, utilizing SHA-1(Secure Hash Algorithm-1, Secure Hash Algorithm) fingerprint algorithm generates the SHA-1 fingerprint value of this document, and this SHA-1 fingerprint value write this document record SHA-1 field.

SHA-1 is a kind of existing fingerprint algorithm, in order to produce can a file of unique identification fingerprint value, and have irreversibility, therefore describe in detail no longer one by one in embodiments of the present invention.

The SHA-1 field writes behind the SHA-1 fingerprint value in order to identify different files.

Whether hit the buffer memory field, in the middle of being identified at historical fingerprint buffer memory storehouse, whether have the history file record that identical SHA-1 fingerprint value is arranged with current file;

If have, then the value of this field is for being; If the history file record of identical SHA-1 fingerprint value with current file does not then write not in field;

Whether whether detected field in order to have determined to exist in the historical record that identifies current file threatens;

Whether belong to when and to threaten field for or not empty the time, whether the value of detected field is for being, otherwise is to deny.

Whether belong to the threat field, in order to identify whether contain threat data in the current file; If have, then the value of this field is for being, if do not have, then the value of this field is for denying;

Whether execution clearance operation field was before let pass in order to log file; Let pass, then being designated as is to deny otherwise be designated as.

Whether retrieves historical fingerprint buffer memory exists the history file that has identical SHA-1 field value with current file in the middle of the storehouse.

Retrieve in the database of historical fingerprint base according to the SHA-1 fingerprint value in the file record, and according to the result for retrieval record that fills up a document.

If there is the history file that records identical SHA-1 fingerprint value with current file, the file that current detection then is described detected in formerly the testing process, therefore, then the value of whether hitting the buffer memory field of described current file being written as of this document be, shows the history file that has hit in the middle of some buffer memorys.

If threaten field not for empty whether belonging in the described history file record, whether the detected field in then described current file being recorded is written as and is, otherwise writes not.If previous this document was tested, then no matter whether belong to and threaten field for being or not, only otherwise be sky, then all illustrate and grasped the information whether this document contains threat data.

Whether the value that threatens field that whether belongs to that the value that threatens field writes described current file that whether belongs to in the described history file record belongs to the threat field and whether contains threat data in order to identify current file.

The described clearance operational word segment value of whether carrying out just can be inserted record only after being let pass or blocking, and is empty in the process of execution in step S220.

Do not have the history file that has identical SHA-1 fingerprint value with current file, then the buffer memory field of whether hitting with described current file is written as not, and showing in the middle of the history buffer storehouse does not have identical SHA-1 fingerprint value record.Simultaneously whether described detected field not being write not, will describedly whether belong to the threat field writes sky, whether carries out the clearance operation field and write sky.

If in historical fingerprint buffer memory storehouse comparison, when hitting, inherit in the historical fingerprint buffer memory storehouse whether belong to the threat field value, and should record the result and sent to described threat strategy module 4.If in historical fingerprint buffer memory storehouse, compare, do not hit, then will whether hit the buffer memory field value and write not, whether belong to the threat field value and continue as sky, then this record is sent to described threat strategy module 4.

In described step S220 implementation, three options are arranged when facing the file of P2P agreement, one is that the SHA-1 fingerprint value that provides among the P2P is provided fully, and a kind of is that the SHA-1 fingerprint value that provides among the P2P is provided fully, and another is to carry out trust authentication to the SHA-1 fingerprint value.

When selecting to trust fully, the SHA-1 fingerprint value that provides in the use P2P agreement is not detained data as the SHA-1 signature of file; When selecting trust authentication, the SHA-1 fingerprint value that provides in the P2P agreement then is provided, and use this SHA-1 fingerprint value in threatening historical judge module, to judge, after the judged result generation, also require data detection module after having loaded file data, last packet of suppressing a document waits the SHA-1 fingerprint value that is ready to use in spanned file to generate and finishes.If the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is identical, then last packet is let pass, if the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is inequality, then use newly-generated file SHA-1 fingerprint value threatening historical judge module to carry out fingerprint threat judgement; When selecting to distrust fully, then make the SHA-1 fingerprint value of spanned file carry out the judgement of history threat fingerprint.

Step S230. works as described cache optimization module and is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for being, when whether belonging to threat field non-NULL, being that index finishes described data detection module to the detection of current file record institute respective file with the main thread task ID in this document record.

If whether the detected words segment value of the file that returns record for being, illustrate that the file of current detection is formerly through detecting this moment, therefore need not again this document to be scanned contrast one by one, be that index stops relative threat detection task with the mission number, namely end the task of described data detection module 3.

When described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for not the time, after waiting for that then described data detection module finishes the detection of current file record respective file, whether what the judged result that described threat strategy unit is returned write the current file record belongs to the threat field.

File whether hit the buffer memory field for not, illustrate that then this document had not before detected, so whether it belongs to threat data or comprises threat data and just need be determined by the judged result of data detection module 3.

When step S240. returned judged result to described threat strategy module when described data detection module prior to described cache optimization module, then described threat strategy unit finished described cache optimization module about the execution in step of described judged result respective file.

Described data detection module is less prior to the probability of described cache optimization module return information, after described data detection module determines whether current file comprises threat data, then ends the task executions of described cache optimization module.

Step S210 and step S220 executed in parallel, when wherein having a thread can determine whether to include threat data in the described detected file, then end the sub-thread of another one of identical file to discharge computational resource as early as possible, such preemption mechanism can reduce the use to computational resource effectively.

The task management data that the described threat strategy resume module of step S250. is relevant, and according to judged result and described threat strategy that described threat judge module 33 returns, carry out forwarding or the blocking-up of data.

The main line task management tabulation that described threat strategy module receiving data stream amount processing module 2 is imported into, storage main line task management tabulation, the signature of the file that the judge module 33 of reception threat simultaneously imports into and task management data, the file record that threatens historical judge module 63 to import into, set up index with the main line mission number in the signature of file, insert sub-line mission number and sub-line task module numbering, insert the task completion status.

Threat strategy according to the user arranges carries out blocking-up or the forwarding of data.If data belong to the threat category, and be set at blocking-up by the user inside threat strategy, then with the data blocking-up, current data can be dropped, and records related data information, and the data subsequent segment is directly abandoned.

If data belong to the threat category, and inside threat strategy, be set at clearance by the user, then forward the data to data and transmit in the middle of the register cell.If data do not belong to the threat category, no matter in the threat strategy the inside, threat is set at blocking-up by the user or does not block, and then also can be forwarded to data and transmit in the middle of the register cell.

Transmit in the register cell in data, passed through the data of detection by the storage size storage, and be transmitted to second network interface card.

Need to prove; the purpose that the present invention arranges cache optimization module 6 is the previous file that detected is no longer carried out secondary detection; and judge whether file had before detected; it mainly is the SHA-1 fingerprint value by file; it is each file that detected through described data detection module 3; all should calculate its SHA-1 fingerprint value in detected back; the step of this calculating can be implemented by the threat judge module in the data detection module 3; as another kind of execution mode; also can be to calculating behind the whole loadeds of file in threat strategy module 4; those skilled in the art can suitably adjust and revise according to the general idea of technical scheme of the present invention these type of ins and outs, but all should be encompassed within the scope that the present invention protects.

When step S260. does not have the SHA-1 value in the file record of current file, then calculate the SHA-1 fingerprint value of current file, and the described judged result that whether belongs to threat data that the SHA-1 fingerprint value that calculates and described threat judge module send is sent to described historical fingerprint buffer memory storehouse, described historical fingerprint buffer memory storehouse is carried out the historical information record and is upgraded and storage.In the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert simultaneously, file SHA-1 value, whether belong to threats (be/not), system's detection time (Year/Month/Day), the fingerprint classification that detects (virus, spam, network address are classified, code injection), detection fingerprint base version number, other descriptions (sky).

The necessary condition of calculating the SHA-1 value is to grasp complete file, because being divided into a plurality of data segments in complete file of data detection module transmits in real time, therefore file can appear through after detecting, do not calculated the situation of SHA-1 value, therefore, when not having the SHA-1 value in the file record of file, then calculate the SHA-1 value of current file.Again described judged result and described SHA-1 fingerprint value are back in the middle of the described historical fingerprint buffer memory storehouse in the lump.Described historical fingerprint buffer memory storehouse transaction file record, in the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert file SHA-1 fingerprint value, fingerprint classification (virus, spam, network address classification, code inject), detection fingerprint base version number, other descriptions (sky) of whether belonging to threat (be/not), system's detection time (Year/Month/Day), detecting.

Preferably, the information of also whether current file being let pass returns to described historical fingerprint buffer memory storehouse in the lump herein, whether carries out the clearance operation field in order to write, and records the information whether current file was before let pass.

If the user does not wish to block any threat, namely threat strategy is set to let pass, and data are transmitted register cell when transmitting threat to the user, can send a piece of news, inform the title of user file title and virus, reminds the user not open this document.

At the second network interface card place, will transmit the data that register cell passes over by data and send to the user by Transparent Proxy according to the TCP/IP mode.If the total data of a file all is handed down to the user, the user can normally use.If wherein a part is handed down to the client, when further part was blocked, the user only can take incomplete data, and can not be performed.

Need to prove, when in the data traffic processing module, having by the file of complete loading, also can carry out the SHA-1 value to it calculates, namely can be carried out by the module that any one of device of the present invention or system grasped the complete information of file the calculating of SHA-1 value, above-described embodiment only is that the embodiment to technical scheme of the present invention is illustrated, and can not be interpreted as whole execution mode of the present invention.

In the present invention, used the SHA-1 algorithm to identify the file uniqueness, SHA-1 is to the input of length less than 264 bits (bit), and producing length is the hashed value of 160 bits (bit), and therefore anti-exhaustive (brute-force) property is better.

In recent years, be proved to be the existence of the possibility of logic conflict based on the fingerprint algorithm of MD5, can have caused different file contents to have identical MD5 fingerprint by the prefix form algorithm.So the fingerprint algorithm of MD5 can not guarantee to judge that correctly file is unique that therefore, the present invention does not use the MD5 fingerprint to carry out the uniqueness identification of file.

But; need to prove method provided by the present invention; though advocating application SHA-1 algorithm generation fingerprint retrieves; but it was obvious that; also can generate MD5 fingerprint or other fingerprints to each file by other algorithms such as MD5 fingerprints; and in the middle of the historical fingerprint buffer memory storehouse of the historical record that includes MD5 fingerprint or other fingerprints, retrieve; those skilled in the art can technical scheme according to the present invention replace the SHA-1 fingerprint value with other fingerprint algorithms, also should belong to protection scope of the present invention.

This method is carried out task management by rational regulation mechanism to data detection module 3 and cache optimization module 6, has realized the lifting of processor performance, has reduced the load of invalid computing.

The testing process that data detection module of the present invention 3 is performed, segmentation sends the technological means of the unit data of having checked to the user, reduce the time of delay that the user receives required data, in the back in the testing process of segmentation, if the message such as file name at user's hostile content place are then interrupted and point out to the discovery hostile content in current location; If the performed steps of cache optimization module 6 find it is the result who had checked, then call in the past history and judge and use current threat strategy, interrupt the performed task of parallel cache optimization module 6 simultaneously.

Like this, user's data downloaded is guaranteed safety and is not had the data of malicious codes such as virus that the handling property that has accelerated testing process has greatly improved time performance and processing speed.Reduce the performance consumption of duplicate detection, obtained huge raising at aspect of performance.

Simultaneously, the file fingerprint that trusting the P2P agreement provides carries out historical scan-data fingerprint contrast and also effectively improves processing speed and save computational resource, but to carry out the conforming comparison of fingerprint before in the end a packet issues, can ensure the authenticity of the SHA-1 fingerprint value that the P2P agreement provides like this.If the fingerprint value that contrast discovery P2P agreement provides is different with the final fingerprint value of file, then retrieve again according to the mode of other agreements.

Though carry out the live load that has carried out strengthening a point processor in the time of data detection module 3 and cache optimization module 6, but the task in these two steps will inevitably be carried out in whole detection and transmission course, processor has only increased the field that it is controlled, and data detection module 3 and cache optimization module 6 are carried out in the mode of seizing, if finishing the current task of oneself, either party all can finish also abortive step by threat strategy module 4, thereby accelerated the release to the processor computational resource, improve the idling-resource of processor with this, d/d computational resource can be used for other tasks, thereby has improved processing speed.

If HTTP, FTP, POP3 agreement scanning document, described cache optimization module 6 is finished the work prior to described data detection module 3 certainly, stops the task of described cache optimization module 6 by threat strategy module 4 but described data detection module 3 might be found to threaten when checking.

And during the scanning of P2P agreement, by the SHA1 fingerprint value that information P2P agreement provides, the execution in step that can allow the judgement of described cache optimization module 6 have precedence over data detection module 3 is finished.If there is not historical scanned file, virus might also not finished the detection of data detection module 3 at the end of file when complete download.In case finished the detection of described data detection module 3, can be considered to threaten and handle according to threat strategy, also can record its SHA1 fingerprint value simultaneously, be used for the comparison foundation of later history scanning.But threat strategy module 4 can be set blocking-up or clearance according to the user when this file was identified.

A kind of data detection system end to end provided by the present invention, apparatus and method, by the scanning piecemeal to content, detecting content is each data part, can be owing to adopting sampling the problem of precise decreasing to occur, be to belong to sampling highly accurately, obtaining huge raising aspect the fine-grained fail safe of sampling.And by a history detection retrieving that is parallel to testing process is set, check the file that whether is detecting to detect in preceding process, retrieval by historical record, prevented the repeatability of work, and the SHA-1 fingerprint value that uses comes file of unique identification, more accurate, can not produce mistake or conflict.

Should be noted that at last that obviously those skilled in the art can carry out various changes and modification to the present invention and not break away from the spirit and scope of the present invention.Like this, if of the present invention these revise and modification belongs within the scope of claim of the present invention and equivalent technologies thereof, then the present invention also is intended to comprise these changes and modification.

Claims

1. a data detection system end to end comprises first network interface card and second network interface card, it is characterized in that, also comprises data traffic processing module, data detection module, threat strategy module;

Described data traffic processing module, the data that are used for the inflow of scanning first network interface card detect required necessary information to obtain;

Described data detection module, be used for according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module transmission, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module through the unit data of contrast, corresponding judged result;

Described threat strategy module is used for determining whether described unit data through contrast is forwarded to described second network interface card according to the judged result that described data detection module sends.

2. data detection system end to end according to claim 1 is characterized in that, described data detection module comprises segmentation module, threatens fingerprint base and threatens judge module;

Described segmentation module is used for the data after the described data traffic processing module scanning are carried out segmentation and be transferred to threatening judge module 33;

Described threat fingerprint base is the database of the known threat data of a storage;

Described threat judge module, be used for searching for according to the segment data after the segmentation module segmentation threat data of described threat fingerprint base, confirm whether to have identical with it data segment, will with described threat fingerprint base in the data segment that mates fully of described threat data be judged as threat data, to be judged as secure data with the data segment that the threat data that threatens in the fingerprint base not exclusively mates, and will be sent to described threat strategy module through unit data and the corresponding judged result of contrast.

3. data detection system end to end according to claim 1 is characterized in that, also comprises the cache optimization module;

Be used for each file is generated a file record, described file record comprises file SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in the current file record according to the history file record, and the current file record is sent to described threat strategy module;

Described historical fingerprint buffer memory storehouse is used for storage and upgrades all through the file record of the file of detection.

4. data detection system end to end according to claim 3, it is characterized in that, described data traffic processing module, also be used for setting up main line task management tabulation, be responsible for the subtask management of data detection module and cache optimization module, and main line task management tabulation is sent to the threat strategy module, again the main thread mission number is sent to data detection module and cache optimization module.

5. data detection system end to end according to claim 4, it is characterized in that, described cache optimization module also is used for according to the unit data that belongs to complete file that receives, make up sub-line mission number, and module numbering, main line mission number, sub-line mission number and task status task management data are sent to described threat strategy module.

6. data detection system end to end according to claim 4, it is characterized in that, described data detection module also is used for the main line mission number according to the data traffic processing module, make up sub-line mission number and sub-line task module numbering, after handling whole unit datas, to each complete file according to the SHA-1 algorithm, produce the signature of each file, described signature comprises settling time, the SHA-1 fingerprint value, the main line mission number, sub-line task module numbering, sub-line mission number and task status.

7. data detection system end to end according to claim 5 is characterized in that, described cache optimization module comprises generation logging modle, historical fingerprint buffer memory storehouse and threatens historical judge module;

Described generation logging modle, be used for according to the unit data that belongs to complete file that receives, make up sub-line mission number, and sub-line task module numbering, main line mission number, sub-line mission number and task status be sent to described threat strategy module, also be used for the spanned file record, described file record comprise file the SHA-1 field, whether hit the buffer memory field, whether detected field, whether belong to and threaten field and whether carry out the clearance operation field;

The historical judge module of described threat is used for whether having the history file that has identical SHA-1 fingerprint value with current file in the middle of the described historical fingerprint buffer memory of the retrieval storehouse;

If there is the history file that has identical SHA-1 fingerprint value with current file, then the value of whether hitting the buffer memory field of described current file is written as and is; If threaten field for empty whether belonging to during described history file records, then whether the detected field in the described current file record being written as is to deny otherwise write; With the value that threatens field that whether belongs to that the value that threatens field writes described current file that whether belongs in the described history file record; The described clearance operational word segment value of whether carrying out is for empty; Do not have the history file that has identical SHA-1 fingerprint value with current file, then the buffer memory field of whether hitting with described current file is written as not, whether described detected field is write not, the described threat field that whether belongs to is write sky, whether carries out the value of clearance operation field for empty; The current file record is back to described threat strategy module.

8. data detection system end to end according to claim 1, it is characterized in that, described threat strategy module, also be used for determining whether described data through contrast are forwarded to described second network interface card strategy that whether allows threat data to let pass that described threat strategy sets up on their own for the user according to threat strategy.

9. data detection system end to end according to claim 8 is characterized in that, described threat strategy module comprises threat strategy unit, blocking unit and data forwarding register cell;

Described threat strategy unit, be used for the main line task management tabulation that receiving data stream amount processing module is imported into, storage main line task management tabulation, the signature of the file that the judge module of reception threat simultaneously imports into and task management data, the file record that threatens historical judge module to import into, set up index with the main line mission number in the signature of file, insert sub-line mission number and sub-line task module numbering, insert the task completion status, and according to judged result and described threat strategy that described threat judge module returns, carry out forwarding or the blocking-up of data:

When judged result when belonging to secure data, then the data of correspondence are let pass, be sent to described data and transmit register cell; The judged result of returning when described threat judge module then sends a reminder message to the user and does not carry out the threat data that has obtained to remind the user for belonging to threat data and threat strategy when letting pass when transmitting data; When the judged result of returning when described threat judge module was set to block for belonging to threat data and threat strategy, then the data with correspondence were sent to blocking unit;

Described data are transmitted register cell, are used for being forwarded to second network interface card by the data that described threat strategy unit sends;

Described blocking unit is used for and will be abandoned by the threat data that described threat strategy unit sends, and ends the detection to described threat data place file further part, and the data that stop to detect will all abandon.

10. according to claim 3 and 9 described data detection systems end to end, it is characterized in that described threat strategy unit also is used for the described data detection module of management and described cache optimization module:

When described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for being, when whether belonging to threat field non-NULL, being that index finishes described data detection module to the detection of current file record institute respective file with the main thread task ID in this document record;

When described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for not the time, after waiting for that then described data detection module finishes the detection of current file record respective file, whether belong to the threat field with what the judged result of returning write the current file record;

When described data detection module is returned judged result to described threat strategy module prior to described cache optimization module, then finish described cache optimization module about the execution in step of described judged result respective file.

11. data detection system end to end according to claim 9, it is characterized in that, described data are transmitted register cell, also be used for when the file record of current file does not have the SHA-1 value, then calculate the SHA-1 fingerprint value of current file, and the SHA-1 fingerprint value of calculating and the described judged result that whether belongs to threat data of described threat judge module transmission are sent to described historical fingerprint buffer memory storehouse; In the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert simultaneously file SHA-1 fingerprint value, fingerprint classification (virus, spam, network address classification, code injection), detection fingerprint base version number, other descriptions (sky) of whether belonging to threats (be/not), system's detection time (Year/Month/Day), detecting.

12. data detection system end to end according to claim 7 is characterized in that, described generation logging modle also comprises the P2P protocol processing unit;

When described P2P protocol processing unit, the SHA-1 fingerprint value that in the cache optimization module P2P agreement is provided are carried out the judgement of history threat fingerprint, can select three options for use, fully trust, trust authentication, distrust fully;

In the cache optimization module, when selecting to trust fully, the SHA-1 fingerprint value that provides in the use P2P agreement is not detained data as the SHA-1 signature of file;

When selecting trust authentication, the SHA-1 fingerprint value that provides in the P2P agreement then is provided, and use this SHA-1 fingerprint value in threatening historical judge module, to judge, after the judged result generation, also require data detection module after having loaded file data, last packet of suppressing a document waits the SHA-1 fingerprint value that is ready to use in spanned file to generate and finishes; If the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is identical, then last packet is let pass, if the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is inequality, then uses newly-generated file SHA-1 fingerprint value to carry out history and threaten fingerprint to judge;

When selecting to distrust fully, then make the SHA-1 fingerprint value of spanned file carry out the judgement of history threat fingerprint.

13. a data detection device end to end is connected end to end between first network interface card and second network interface card, it is characterized in that, comprises data traffic processing module, data detection module, threat strategy module and cache optimization module;

Described threat strategy module is used for the judged result according to described data detection module transmission, sets with reference to threat strategy to determine whether described unit data through contrast is forwarded to described second network interface card;

Described cache optimization module is used for each file is generated a file record, and described file record comprises file SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in generation current file record according to the history file record, and the current file record is sent to described threat strategy module; Described historical fingerprint buffer memory storehouse is used for storage and upgrades all through the historical information record of the file of detection.

14. a Data Detection method end to end is characterized in that, comprises step:

Steps A. the data that described data traffic processing module scans the inflow of first network interface card detect required necessary information to obtain;

The described data detection module of step B. is according to described necessary information, with the known threat data of search after the data segmentation of described data traffic processing module transmission, confirm whether identical with it data segment is arranged, judge whether there is threat data in the described unit data according to comparing result; And will be sent to described threat strategy module through the unit data of contrast, corresponding judged result;

The judged result that the described threat strategy module of step C. sends according to described data detection module determines whether described unit data through contrast is forwarded to described second network interface card.

15. the method for Data Detection end to end according to claim 14 is characterized in that, among the described step B, describedly judges whether there is threat data in the described unit data, comprise the steps: according to comparing result

Will with described threat fingerprint base in the data segment that mates fully of described threat data be judged as threat data; Will with described threat fingerprint base in threat data not exclusively the data segment of coupling be judged as secure data.

16. the method for Data Detection end to end according to claim 14 is characterized in that described step C also comprises the steps:

Described threat strategy module also determines whether described unit data through contrast is forwarded to described second network interface card according to threat strategy.

17. the method for Data Detection end to end according to claim 16 is characterized in that described step C comprises:

The judged result that step C1. returns when described data detection module is when belonging to secure data, then the unit data with correspondence is forwarded to second network interface card, the judged result of returning when described data detection module is for belonging to threat data and threat strategy when letting pass, then the unit data with correspondence is forwarded to second network interface card, sends reminder message to the user simultaneously and does not open the file at threat data place to remind the user;

When the judged result that step C2. returns when described data detection module was set to block for belonging to threat data and threat strategy, then the data with correspondence abandoned, and ended the detection to the further part of threat data place file.

18. the method for Data Detection end to end according to claim 14 is characterized in that described step B also comprises:

Step B '. the cache optimization module generates a file record to each file, and described file record comprises file SHA-1 fingerprint value at least; The SHA-1 fingerprint value that generates with current file is the history file record that has identical SHA-1 fingerprint value in the historical fingerprint buffer memory of the indexed search storehouse, fill in generation current file record according to the history file record, and the current file record is sent to described threat strategy module.

19. the method for Data Detection end to end according to claim 18, it is characterized in that, described steps A also comprises step: set up main line task management tabulation, be responsible for the subtask management of data detection module and cache optimization module, and main line task management tabulation is sent to the threat strategy module, again the main thread mission number is sent to data detection module and cache optimization module.

20. the method for Data Detection end to end according to claim 19 is characterized in that described step B also comprises step:

According to the main line mission number of data traffic processing module, make up sub-line mission number and sub-line task module numbering, after handling total data unit, each complete file according to the SHA-1 algorithm, is produced the signature of each file.

21. the method for Data Detection end to end according to claim 18 is characterized in that described step B ' also comprises step:

According to the unit data that belongs to complete file that receives, make up sub-line mission number, and module numbering, main line mission number, sub-line mission number and task status task management data are sent to described threat strategy module.

22. the method for Data Detection end to end according to claim 21 is characterized in that described step B ' comprising:

Step B1 '. according to the unit data that belongs to complete file that receives, make up sub-line mission number, and sub-line task module numbering, main line mission number, sub-line mission number and task status are sent to described threat strategy module, spanned file record;

Step B2 '. whether the historical judge module retrieves historical of described threat fingerprint buffer memory exists the history file that has identical SHA-1 fingerprint value with current file in the middle of the storehouse;

There is the history file that has identical SHA-1 fingerprint value with current file, then the value of whether hitting the buffer memory field of described current file is written as and is; If threaten field for empty whether belonging to during described history file records, then whether the detected field in the described current file record being written as is to deny otherwise write; With the value that threatens field that whether belongs to that the value that threatens field writes described current file that whether belongs in the described history file record; The described clearance operational word segment value of whether carrying out is for empty;

Do not have the history file that has identical SHA-1 fingerprint value with current file, then whether the buffer memory field of whether hitting with described current file is not written as not, will described detected field writes not, the described threat field that whether belongs to is write sky, whether carries out the clearance operation field and write sky;

Step B3 '. the historical judge module of described threat is back to described threat strategy module with the current file record.

23. the method for Data Detection end to end according to claim 22 is characterized in that, after the described step B, also comprises before the described step C:

Step C1 '. when described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for being, when whether belonging to threat field non-NULL, being that index finishes described data detection module to the detection of current file record institute respective file with the main thread task ID in this document record;

When described cache optimization module is recorded to described threat strategy module prior to described data detection module backspace file, and whether the value of the described detected field in the current file that described cache optimization module the is returned record for not the time, after waiting for that then described data detection module finishes the detection of current file record respective file, whether what the judged result that described threat strategy unit is returned write the current file record belongs to the threat field;

Step C2 '. when described data detection module was returned judged result to described threat strategy module prior to described cache optimization module, then described threat strategy unit finished described cache optimization module about the execution in step of described judged result respective file.

24. the method for Data Detection end to end according to claim 20 is characterized in that, also comprises after the described step C:

When described data forwarding register cell does not have the SHA-1 value in the file record of current file, then calculate the SHA-1 fingerprint value of current file, and the SHA-1 fingerprint value of calculating and the described judged result that whether belongs to threat data of described threat judge module transmission are sent to described historical fingerprint buffer memory storehouse; In the record sheet that safeguard in described historical fingerprint buffer memory storehouse, insert simultaneously file SHA-1 fingerprint value, fingerprint classification (virus, spam, network address classification, code injection), detection fingerprint base version number, other descriptions (sky) of whether belonging to threats (be/not), system's detection time (Year/Month/Day), detecting.

25. the method for Data Detection end to end according to claim 22 is characterized in that, when the file of first network interface card inflow was the P2P document of agreement, then after described steps A, step B1 ' also comprised before:

Steps A 1 '. the SHA-1 fingerprint value that the cache optimization module provides the P2P agreement carries out history and threatens fingerprint to judge, is provided with three options, fully trust, trust authentication, distrust fully;

Steps A 2 '. in the cache optimization module, when selecting to trust fully, the SHA-1 fingerprint value that provides in the use P2P agreement is not detained data as the SHA-1 signature of file; When selecting trust authentication, the SHA-1 fingerprint value that provides in the P2P agreement then is provided, and use this SHA-1 fingerprint value in threatening historical judge module, to judge, after the judged result generation, also require data detection module after having loaded file data, last packet of suppressing a document waits the SHA-1 fingerprint value that is ready to use in spanned file to generate and finishes; If the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is identical, then last packet is let pass, if the SHA-1 fingerprint value that provides in the SHA-1 fingerprint value that generates and the P2P agreement is inequality, then use newly-generated file SHA-1 fingerprint value threatening historical judge module to carry out fingerprint threat judgement; When selecting to distrust fully, then make the SHA-1 fingerprint value of spanned file carry out the judgement of history threat fingerprint.