CN113271236A - Engine evaluation method, device, equipment and storage medium - Google Patents
Engine evaluation method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113271236A CN113271236A CN202110653724.6A CN202110653724A CN113271236A CN 113271236 A CN113271236 A CN 113271236A CN 202110653724 A CN202110653724 A CN 202110653724A CN 113271236 A CN113271236 A CN 113271236A
- Authority
- CN
- China
- Prior art keywords
- flow
- traffic
- test
- engine
- black
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011156 evaluation Methods 0.000 title claims abstract description 47
- 238000012360 testing method Methods 0.000 claims abstract description 221
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims description 55
- 238000004880 explosion Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 5
- ZXQYGBMAQZUVMI-GCMPRSNUSA-N gamma-cyhalothrin Chemical compound CC1(C)[C@@H](\C=C(/Cl)C(F)(F)F)[C@H]1C(=O)O[C@H](C#N)C1=CC=CC(OC=2C=CC=CC=2)=C1 ZXQYGBMAQZUVMI-GCMPRSNUSA-N 0.000 claims description 4
- 239000000523 sample Substances 0.000 claims 1
- 238000011990 functional testing Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000002457 bidirectional effect Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000011068 loading method Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1441—Countermeasures against malicious traffic
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The embodiment of the invention provides a method, a device, equipment and a storage medium for engine evaluation, wherein the method comprises the following steps: constructing a network flow test set; the network flow test set comprises white flow and black flow; constructing a flow test model based on the network flow test set to form a flow test packet; sending the flow test packet to an engine to be tested, and receiving a test result fed back by the engine to be tested; and comparing the test result with the actual result of the flow test packet, and evaluating the engine to be tested based on the comparison result. The technical scheme provided by the embodiment of the invention can be used for carrying out functional test on the engine, and the test efficiency and precision are improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of internet, in particular to an engine evaluation method, device, equipment and storage medium.
Background
With the rapid development of the internet, network security is also concerned, and analysis and detection aiming at the flow in the network are important means for discovering network security events. Related safety prevention systems are deployed and built in various industries, the systems play a vital role in monitoring network flow in real time and discovering abnormal events such as network attack and the like in time through a flow detection engine, and powerful technical support is provided for maintaining network safety of countries and enterprises and reducing enterprise and personal property loss.
At present, the flow detection engine is irregular in quality, and a means or a product for objectively judging the functional performance of the flow detection engine is lacked. In addition, the flow detection engine is insufficient in some aspects, and often has the defects of dependence on known rules, single algorithm, insufficient detection capability on the events such as vulnerability attacks, domain name generation algorithm (DGA) domain names, malicious code propagation and the like. Therefore, comprehensive evaluation of the flow detection engine is imperative.
Disclosure of Invention
The embodiment of the invention provides an engine evaluation method, an engine evaluation device, engine evaluation equipment and a storage medium, which can be used for performing functional test on an engine and improving the efficiency and the precision of the test.
In a first aspect, an embodiment of the present invention provides a method for engine evaluation, including:
constructing a network flow test set; the network flow test set comprises white flow and black flow; constructing a flow test model based on the network flow test set to form a flow test packet;
sending the flow test packet to an engine to be tested, and receiving a test result fed back by the engine to be tested;
and comparing the test result with the actual result of the flow test packet, and evaluating the engine to be tested based on the comparison result.
In a second aspect, an embodiment of the present invention further provides an engine evaluation apparatus, including:
the construction module is used for constructing a network flow test set; the network flow test set comprises white flow and black flow;
the forming module is used for constructing a flow test model based on the network flow test set to form a flow test packet;
the sending/receiving module is used for sending the flow test packet to an engine to be tested and receiving a test result fed back by the engine to be tested;
and the evaluation module is used for comparing the test result with the actual result of the flow test packet and evaluating the engine to be tested based on the comparison result.
In a third aspect, an embodiment of the present invention provides an electronic device, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the methods provided by the embodiments of the present invention.
In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the method provided by the present invention.
According to the technical scheme provided by the embodiment of the invention, the network flow test set is constructed, the flow test model is constructed on the basis of the network flow test set to form the flow test packet, the flow test packet is sent to the engine to be tested, the test result fed back by the engine to be tested is received, the test result and the actual result of the flow test packet are compared, the engine to be tested is evaluated on the basis of the comparison result, the engine can be functionally tested, and the test efficiency and the test precision are improved.
Drawings
FIG. 1 is a flow chart of a method for engine evaluation according to an embodiment of the present invention;
fig. 2 is a flow chart of a network traffic test set construction according to an embodiment of the present invention;
FIG. 3 is a flow chart of flow test package formation according to an embodiment of the present invention;
fig. 4 is a flowchart of measurement test packet transmission when a traffic test task is created according to an embodiment of the present invention;
FIG. 5 is a flow chart of test result evaluation provided by an embodiment of the present invention;
FIG. 6 is a block diagram of an engine evaluation apparatus according to an embodiment of the present invention;
fig. 7 is a block diagram of a flow detection platform according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a flowchart of a method for engine evaluation according to an embodiment of the present invention, where the method may be performed by an engine evaluation apparatus, where the apparatus may be implemented by software and/or hardware, and the apparatus may be configured in a traffic detection platform, where the traffic detection platform provides a test environment compatible with a traffic detection engine carried by a server for performing a functionality test, and the method may be applied in a scenario where the traffic detection engine is evaluated.
As shown in fig. 1, the technical solution provided by the embodiment of the present invention includes:
s110: constructing a network flow test set; the network traffic test set comprises white traffic and black traffic.
In an implementation manner of the embodiment of the present invention, the constructing a network traffic test set includes: simulating black traffic based on the network security event type, and storing the black traffic; wherein, the black flow is marked according to the size and the type, the hazard degree, the detection difficulty and the outbreak year; simulating white flow and storing the white flow; the white traffic comprises fixed network white traffic and mobile white traffic; and forming a flow test set by the stored real-time flow, the black flow and the white flow. The network traffic test set may further include real-time traffic, where the real-time traffic may be traffic obtained from various real network environments. Optionally, the network security event types include detection scanning, malicious connection, vulnerability attack, trojan control, malicious code downloading, phishing mails, web backdoors, and malicious domain name links. The network security event type is not limited to the above event type, and may include other event types.
In the embodiment of the invention, the background flow in the flow test set can be mainly fixed network white flow; since the performance of the engine to be tested is mainly evaluated based on the detection related to the security event, the flow test set mainly attacks the black flow. The white traffic in the traffic test set may include access traffic and response traffic, and the black traffic may include attack traffic and response traffic. When testing the engine to be tested, the unidirectional flow of the white flow or the black flow can be used independently, and the bidirectional flow can also be used.
In the embodiment of the invention, white flow, real-time flow and various black flows can be added into the flow library to form a flow test set, wherein labels such as classification, difficulty in detection, degree of harm, outbreak timeliness and the like are required to be marked on the black flows. Different types of traffic are stored in different directories.
S120: and constructing a flow test model based on the network flow test set to form a flow test packet.
In the embodiment of the invention, the flow test model can be constructed on the basis of the network event type customized broad spectrum standard test set based on the engine to be tested, and the flow test model can be understood as different flows selected from the flow library according to the flow test task to form the flow test packet.
In an implementation manner of the embodiment of the present invention, optionally, the constructing a traffic test model based on the network traffic test set to form a traffic test packet includes: selecting white traffic and black traffic from the network traffic test set based on a preset proportion; determining the traffic test packet based on the selected white traffic and black traffic; wherein, the black flow is selected according to the attribute proportion.
In the embodiment of the invention, the flow test packet should be mixed with various types of black flow and white flow, and also needs to cover the types of hazard degree, detection difficulty, explosion time and the like. In an implementation manner of the embodiment of the present invention, optionally, the determining the traffic test packet based on the selected white traffic and the black traffic includes: recombining the selected white flow and black flow; randomizing the source address or the destination address of the white flow and the black flow; and forming a flow test packet by the processed white flow, real-time flow and black flow.
Specifically, the white traffic and the black traffic may be set according to the traffic test task, and the white traffic and the black traffic may be selected from the network traffic test set in the library according to a preset ratio. The black flow can be selected in proportion according to attributes such as type, explosion timeliness, hazard degree and detection difficulty. The white traffic and the black traffic are recombined, specifically, source addresses and destination addresses of the white traffic and the black traffic are randomized, so that an irregular traffic test packet can be formed, and an engine to be tested can be better detected.
S130: and sending the flow test packet to an engine to be tested, and receiving a test result fed back by the engine to be tested.
In the embodiment of the present invention, relevant parameters of traffic transmission may be read from a configuration set by a traffic detection platform, a transmission rate, a start time, a size of transmitted traffic, and a transmission frequency may be selected, and a suitable port may be selected to transmit a test traffic packet, where the traffic test packet may be simultaneously transmitted to a plurality of engines to be tested by an optical splitter as needed. Wherein, the engine under test can be a flow detection engine.
In the embodiment of the invention, the engine to be tested receives the flow test packet, monitors and identifies the white flow and the black flow in the flow test packet, and sends the test result to the flow detection platform, and the flow detection platform receives the test result. And the test result comprises the monitoring result of the white flow and the black flow in the flow test packet.
S140: and comparing the test result with the actual result of the flow test packet, and evaluating the engine to be tested based on the comparison result.
In the embodiment of the invention, in the test process, the flow detection platform can receive the test result fed back by the engine to be tested through the result reporting interface. The flow detection platform can perform normalization processing on logs of different types of test results and compare the test results with actual results of the flow test packets. In an implementation manner of the embodiment of the present invention, optionally, comparing the test result with the actual result of the network test model, and evaluating the engine to be tested based on the comparison result includes: comparing the test result with the actual result, and determining an objective evaluation score based on the comparison result; determining an assessment total score based on the objective assessment score and the subjective assessment score; and evaluating the engine to be tested based on the evaluation total score.
In an implementation manner of the embodiment of the present invention, optionally, determining an objective evaluation score based on the comparison result includes: determining packet loss rate, accuracy rate, recall rate and false alarm rate based on the comparison result; determining a comprehensive F value based on the accuracy rate and the recall rate; and determining a corresponding objective evaluation score based on the packet loss rate, the accuracy rate, the recall rate, the false alarm rate and the comprehensive F value.
In this embodiment, the packet loss ratio is 1- (total amount of received single-batch traffic test packets/total amount of actually transmitted traffic test packets in the batch). The accuracy rate is the number of correct results in the reported results/the total number of reported results; the recall rate is the number of correct results/total number of correct results in the reported results; the false alarm rate is the number of incorrect results in the reported result/the total number of the reported results; the rate of missing reports is 1-recall rate; the integrated F value is 2 × accuracy × recall/(accuracy + recall). The packet loss rate, the accuracy rate, the recall rate, the false alarm rate and the comprehensive F value respectively correspond to objective evaluation scores, namely the product of the packet loss rate, the accuracy rate, the recall rate, the false alarm rate and the comprehensive F value and a preset numerical value. The subjective evaluation score can be determined based on performance in the test process and special options in the test result, the total evaluation score is determined through the subjective evaluation score and the objective evaluation score, and the engine to be tested is evaluated through the total evaluation score.
On the basis of the foregoing embodiment, the method provided in the embodiment of the present invention may further include: and forming a history record of the test process for inquiry. Therefore, by forming the history record, the system can be used for inquiring, forensics, disputing and refereeing and the like.
In an implementation manner of the embodiment of the present invention, the real-time traffic may be individually sent to the engine to be tested as a traffic test packet, the engine to be tested feeds back a test result of the real-time traffic, and the test result is compared with an actual result of the real-time traffic, so that the engine to be tested is evaluated based on the comparison result.
The embodiment passes through the flow detection platform, can detect the flow detection engine, can know the technical characteristics of each flow detection engine, find out the not enough of its detection ability, fully excavate the function and the performance limit of flow detection engine, explore more and be fit for in different network levels, efficient flow detection scheme under the different environment, the functional standard test standard that provides unified to the flow detection engine, can objectively reflect the technical characteristics and the ability not enough of each flow detection engine, thereby provide technical basis for the limit of the function and the performance of follow-up excavation flow detection engine better provides the network security guarantee, simultaneously can carry out the reference of flow detection engine type selection to network security as enterprise units of all levels.
According to the technical scheme provided by the embodiment of the invention, the network flow test set is constructed, the flow test model is constructed on the basis of the network flow test set to form the flow test packet, the flow test packet is sent to the engine to be tested, the test result fed back by the engine to be tested is received, the test result and the actual result of the flow test packet are compared, the engine to be tested is evaluated on the basis of the comparison result, the engine can be functionally tested, and the test efficiency and the test precision are improved.
Fig. 2 is a flow chart of a network traffic test set construction provided in an embodiment of the present invention, where the network traffic test set is stored in a traffic library, and the traffic library needs to cover but is not limited to a plurality of security event types such as detection scanning, malicious connection, vulnerability attack, trojan control, malicious code downloading, phishing mail, web backdoor, and malicious domain name linking. The white traffic in the traffic test set may include access traffic and response traffic, and the black traffic may include attack traffic and response traffic. When testing the engine to be tested, the unidirectional flow of the white flow or the black flow can be used independently, and the bidirectional flow can also be used.
White flow, real-time flow and various black flows can be added into the flow library, and attribute labels such as labeling classification, detection difficulty, hazard degree and explosion timeliness are required for the black flows. Different types of traffic are stored in different directories and different types of traffic are stored in different directories.
As shown in fig. 2, the network traffic test set construction method is as follows:
and step 101, storing the real-time flow in a warehouse.
And step 102, storing the white traffic in a warehouse.
And step 103, storing the black flow in a warehouse.
The real-time flow is recorded and stored in a flow library, so that no marking is needed, and the white flow is uploaded to the flow library through offline or File Transfer Protocol (FTP) and other modes without marking; the black flow is uploaded to a flow library through offline or FTP and the like, and the black flow needs to be labeled according to attributes (dimensions) such as type, hazard degree, detection difficulty, explosion timeliness and the like, and a white flow, a real-time flow and the black flow in the flow library form a flow test set.
Fig. 3 is a flowchart of flow test packet formation according to an embodiment of the present invention, where in this embodiment, optionally, the flow test packet should mix various types of black flow and white flow, and further needs to cover types such as a hazard level, a detection difficulty, and an explosion time.
As shown in fig. 3, the method for forming the traffic test packet may specifically include:
step 310: and receiving the set flow customization rule.
Step 320: the distribution ratio of each flow rate is calculated, including the white flow rate and the black flow rate.
Step 330: white flow extraction is performed from the measurement test set.
Step 340: and proportionally extracting black flow from the flow test set according to dimensions such as category, effective time, hazard degree, detection difficulty and the like.
Step 350: and carrying out recombination treatment on various flows.
Step 360: the source/destination IP of the white traffic and the black traffic are randomized.
And finally, returning an indication that the creation of the flow test packet is completed in the flow detection platform.
Fig. 4 is a flowchart of sending a measurement test packet when a traffic test task is created according to an embodiment of the present invention, where the test traffic packet can implement unidirectional and bidirectional playback, and the size, frequency, and mixing manner of traffic can be freely adjusted according to actual situations. As shown in fig. 4, the specific steps of sending the traffic test packet include:
step 410: and reading related parameters of flow sending from the configuration set by the flow detection platform, and selecting a sending rate.
Step 420: and reading related parameters of flow sending from the configuration set by the flow detection platform, and selecting the starting time and the ending time.
Step 430: and reading related parameters sent by the flow from the configuration set by the flow detection platform, and selecting the flow.
Step 440: and reading related parameters of flow sending from the configuration set by the flow detection platform, and selecting flow sending frequency.
Step 450: and selecting a port of the server to send a test flow packet, wherein the flow test packet can be connected with the optical splitter and simultaneously sent to a plurality of engines to be tested for use as required.
Fig. 5 is a flowchart of test result evaluation according to an embodiment of the present invention, in which a flow detection platform may perform fast standardized processing on a test result output by a test of an engine to be tested, extract key fields in the test result, calculate a false alarm rate and a false negative rate, may replay the flow magnitude, frequency and a mixed manner for many times according to a test standard, and perform comprehensive scoring and single analysis on the performance and function of the engine to be tested according to a comprehensive determination standard. As shown in fig. 5, the specific steps of test result evaluation include:
step 510: and in the test process, receiving a test result generated by the engine to be tested through the result reporting interface.
Step 520: and carrying out normalization processing on logs of different types of test results through data extraction, conversion and loading (ETL).
Step 530: and comparing the information of the flow test packet with the test result of the engine to be tested, and evaluating and scoring.
Step 540: respectively calculating performance indexes such as false alarm rate, missing report rate, packet loss rate and the like, and calculating corresponding one-way objective evaluation scores.
Step 550: and generating a test report on the flow detection platform and displaying a query on an interface.
Fig. 6 is a block diagram of an engine evaluation apparatus configured in a traffic detection platform according to an embodiment of the present invention, where the apparatus includes: a building module 610, a forming module 620, a sending/receiving module 630, and an evaluation module 640.
The building module 610 is configured to build a network traffic test set; the network flow test set comprises white flow and black flow;
a forming module 620, configured to construct a traffic test model based on the network traffic test set, and form a traffic test packet;
a sending/receiving module 630, configured to send the traffic test packet to an engine to be tested, and receive a test result fed back by the engine to be tested;
an evaluation module 640, configured to compare the test result with the actual result of the traffic test packet, and evaluate the engine to be tested based on the comparison result.
Optionally, the constructing a traffic test model based on the network traffic test set to form a traffic test packet includes:
selecting white traffic and black traffic from the network traffic test set based on a preset proportion;
determining the flow test packet based on the selected white flow, real-time flow and black flow; wherein, the black flow is selected according to the attribute proportion.
Optionally, the determining the traffic test packet based on the selected white traffic, the real-time traffic, and the black traffic includes:
recombining the selected white flow and black flow;
randomizing the source address or the destination address of the white flow and the black flow;
and forming a flow test packet by the processed white flow and the processed black flow.
Optionally, the constructing a network traffic test set includes:
simulating black traffic based on the network security event type, and storing the black traffic; wherein the black flow is marked according to the hazard degree, the detection difficulty and the explosion time;
simulating white flow and storing the white flow; the white traffic comprises fixed network white traffic and mobile white traffic;
and forming the stored black flow and white flow into a flow test set.
Optionally, the network security event types include detection scanning, malicious connection, vulnerability attack, trojan horse control, malicious code downloading, phishing mails, web backdoors, and malicious domain name links.
Optionally, comparing the test result with the actual result of the network test model, and evaluating the engine to be tested based on the comparison result, including:
comparing the test result with the actual result, and determining an objective evaluation score based on the comparison result;
determining an assessment total score based on the objective assessment score and the subjective assessment score;
and evaluating the engine to be tested based on the evaluation total score.
Optionally, determining an objective evaluation score based on the alignment result includes:
determining packet loss rate, accuracy rate, recall rate and false alarm rate based on the comparison result;
determining a comprehensive F value based on the accuracy rate and the recall rate;
and determining a corresponding objective evaluation score based on the packet loss rate, the accuracy rate, the recall rate, the false alarm rate and the comprehensive F value.
The device can execute the method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 7 is a block diagram of a flow detection platform according to an embodiment of the present invention, and as shown in fig. 7, the entire platform is composed of a data storage module, a service processing module, a human-computer interaction module, and an interface module, and mainly implements service processing through the service processing module, but needs to provide a task customized by the human-computer interaction module, and cooperates with the human-computer interaction module to implement, display, and perform fault processing.
The main functions of the platform include:
(1) the access of real-time flow is realized in an interface mode and is used as a part of a flow test set;
(2) the method comprises the following steps of leading in an autonomously set flow test set to an engine to be tested in an interface mode;
(3) receiving a test result of the engine to be tested in an interface mode;
(4) a flow database of white flow and various black flows is stored in the flow detection platform;
(5) the flow detection platform interface can perform autonomous configuration on flow composition and test tasks.
(6) The flow detection platform interface can display the test result evaluation and various targeted statistics of each engine to be tested.
(7) The flow detection platform has certain management functions of user login, log retention, running state monitoring and the like.
Table 1 is an interface function corresponding to the human-computer interaction module, where the interface includes a classification menu such as user management, system configuration, test result evaluation, system log, and system monitoring, and the implemented function may refer to table 1.
TABLE 1
Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 8, the electronic device includes:
one or more processors 810, one processor 810 being illustrated in FIG. 8;
a memory 820;
the apparatus may further include: an input device 830 and an output device 840.
The processor 810, the memory 820, the input device 830 and the output device 840 of the apparatus may be connected by a bus or other means, for example, in fig. 8.
The memory 820, which is a non-transitory computer-readable storage medium, may be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to an engine evaluation method in an embodiment of the present invention (e.g., the building module 610, the forming module 620, the transmitting/receiving module 630, and the evaluating module 640 shown in fig. 6). The processor 810 executes various functional applications and data processing of the computer device by executing software programs, instructions and modules stored in the memory 820, namely, an engine evaluation method for implementing the above method embodiments, that is:
constructing a network flow test set; the network flow test set comprises white flow and black flow;
constructing a flow test model based on the network flow test set to form a flow test packet;
sending the flow test packet to an engine to be tested, and receiving a test result fed back by the engine to be tested;
and comparing the test result with the actual result of the flow test packet, and evaluating the engine to be tested based on the comparison result.
The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the computer device, and the like. Further, the memory 820 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 820 may optionally include memory located remotely from processor 810, which may be connected to the terminal device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 830 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus. The output device 840 may include a display device such as a display screen.
An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements an engine evaluation method according to an embodiment of the present invention:
constructing a network flow test set; the network flow test set comprises white flow and black flow; constructing a flow test model based on the network flow test set to form a flow test packet;
sending the flow test packet to an engine to be tested, and receiving a test result fed back by the engine to be tested;
and comparing the test result with the actual result of the flow test packet, and evaluating the engine to be tested based on the comparison result.
Any combination of one or more computer-readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A method of engine evaluation, comprising:
constructing a network flow test set; the network flow test set comprises white flow, real-time flow and black flow;
constructing a flow test model based on the network flow test set to form a flow test packet;
sending the flow test packet to an engine to be tested, and receiving a test result fed back by the engine to be tested;
and comparing the test result with the actual result of the flow test packet, and evaluating the engine to be tested based on the comparison result.
2. The method of claim 1, wherein constructing a traffic test model based on the network traffic test set, forming a traffic test packet, comprises:
selecting white traffic and black traffic from the network traffic test set based on a preset proportion;
determining the traffic test packet based on the selected white traffic and black traffic; wherein, the black flow is selected according to a plurality of attribute proportions.
3. The method of claim 2, wherein determining the traffic test packet based on the selected white traffic, real-time traffic, and black traffic comprises:
recombining the selected white flow and black flow;
randomizing the source address or the destination address of the white flow and the black flow;
and forming a flow test packet by the processed white flow, real-time flow and black flow.
4. The method of claim 1 or 2, wherein the constructing the network traffic test set comprises:
simulating black traffic based on the network security event type, and storing the black traffic; wherein, the black flow is marked according to the size and the type, the hazard degree, the detection difficulty and the explosion timeliness;
simulating white flow and storing the white flow; the white traffic comprises fixed network white traffic and mobile white traffic;
and forming a flow test set by the stored real-time flow, the black flow and the white flow.
5. The method of claim 4, wherein the network security event types include probe scans, malicious connections, vulnerability attacks, trojan horse control, malicious code downloads, phishing, web backdoors, and malicious domain name links.
6. The method of claim 1, wherein comparing the test result with an actual result of the network traffic test packet, and evaluating the engine under test based on the comparison result comprises:
comparing the test result with the actual result, and determining an objective evaluation score based on the comparison result;
determining an assessment total score based on the objective assessment score and the subjective assessment score;
and evaluating the engine to be tested based on the evaluation total score.
7. The method of claim 5, wherein determining an objective assessment score based on the alignment results comprises:
determining packet loss rate, accuracy rate, recall rate and false alarm rate based on the comparison result;
determining a comprehensive F value based on the accuracy rate and the recall rate;
and determining a corresponding objective evaluation score based on the packet loss rate, the accuracy rate, the recall rate, the false alarm rate and the comprehensive F value.
8. An engine evaluation apparatus, comprising:
the construction module is used for constructing a network flow test set; the network flow test set comprises white flow and black flow;
the forming module is used for constructing a flow test model based on the network flow test set to form a flow test packet;
the sending/receiving module is used for sending the flow test packet to an engine to be tested and receiving a test result fed back by the engine to be tested;
and the evaluation module is used for comparing the test result with the actual result of the flow test packet and evaluating the engine to be tested based on the comparison result.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110653724.6A CN113271236A (en) | 2021-06-11 | 2021-06-11 | Engine evaluation method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110653724.6A CN113271236A (en) | 2021-06-11 | 2021-06-11 | Engine evaluation method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113271236A true CN113271236A (en) | 2021-08-17 |
Family
ID=77234879
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110653724.6A Pending CN113271236A (en) | 2021-06-11 | 2021-06-11 | Engine evaluation method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113271236A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346372A (en) * | 2013-07-31 | 2015-02-11 | 国际商业机器公司 | Method and device for evaluating prediction models |
| CN109685089A (en) * | 2017-10-18 | 2019-04-26 | 北京京东尚科信息技术有限公司 | The system and method for assessment models performance |
| CN109936582A (en) * | 2019-04-24 | 2019-06-25 | 第四范式（北京）技术有限公司 | Construct the method and device based on the PU malicious traffic stream detection model learnt |
| CN110210294A (en) * | 2019-04-23 | 2019-09-06 | 平安科技（深圳）有限公司 | Evaluation method, device, storage medium and the computer equipment of Optimized model |
-
2021
- 2021-06-11 CN CN202110653724.6A patent/CN113271236A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346372A (en) * | 2013-07-31 | 2015-02-11 | 国际商业机器公司 | Method and device for evaluating prediction models |
| CN109685089A (en) * | 2017-10-18 | 2019-04-26 | 北京京东尚科信息技术有限公司 | The system and method for assessment models performance |
| CN110210294A (en) * | 2019-04-23 | 2019-09-06 | 平安科技（深圳）有限公司 | Evaluation method, device, storage medium and the computer equipment of Optimized model |
| CN109936582A (en) * | 2019-04-24 | 2019-06-25 | 第四范式（北京）技术有限公司 | Construct the method and device based on the PU malicious traffic stream detection model learnt |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11750659B2 (en) | Cybersecurity profiling and rating using active and passive external reconnaissance | |
| US10754958B1 (en) | Vulnerability risk mitigation platform apparatuses, methods and systems | |
| US10027705B1 (en) | Apparatuses, methods and systems for a real-time cyber threat indicator verification mechanism | |
| CN112003838B (en) | Network threat detection method, device, electronic device and storage medium | |
| CN105376245A (en) | Rule-based detection method of ATP attack behavior | |
| US20220014561A1 (en) | System and methods for automated internet-scale web application vulnerability scanning and enhanced security profiling | |
| CN112905548B (en) | Security audit system and method | |
| CN104182681B (en) | Hook-based iOS (iPhone operating system) key behavior detection device and detection method thereof | |
| CN113315767B (en) | Electric power internet of things equipment safety detection system and method | |
| CN103581185A (en) | Cloud searching and killing method, device and system for resisting anti-antivirus test | |
| CN111176202A (en) | Safety management method, device, terminal equipment and medium for industrial control network | |
| CN115733646A (en) | Network security threat assessment method, device, equipment and readable storage medium | |
| CN113987509A (en) | Risk rating method, device, equipment and storage medium for information system security vulnerability | |
| CN116451215A (en) | Correlation analysis method and related equipment | |
| CN110830500B (en) | Network attack tracking method and device, electronic equipment and readable storage medium | |
| CN115310090A (en) | Terminal reliability dynamic detection system | |
| CN114125083A (en) | Industrial network distributed data acquisition method and device, electronic equipment and medium | |
| WO2021243555A1 (en) | Quick application test method and apparatus, device, and storage medium | |
| CN112528295A (en) | Vulnerability repairing method and device of industrial control system | |
| WO2021130897A1 (en) | Analysis device, analysis method, and non-transitory computer-readable medium storing analysis program | |
| CN113271236A (en) | Engine evaluation method, device, equipment and storage medium | |
| CN114866434B (en) | Network asset security assessment method and application | |
| CN115525897A (en) | System detection method and device for terminal equipment, electronic device and storage medium | |
| WO2014048194A1 (en) | Android malicious application program detection method, system and device | |
| CN113824736A (en) | Asset risk handling method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210817 |