CN116010228B

CN116010228B - Time estimation method and device for network security scanning

Info

Publication number: CN116010228B
Application number: CN202310279637.8A
Authority: CN
Inventors: 罗泽龙; 韩冰; 李可; 李季
Original assignee: Beijing Yuanbao Technology Co ltd
Current assignee: Beijing Yuanbao Technology Co ltd
Priority date: 2023-03-22
Filing date: 2023-03-22
Publication date: 2023-06-30
Anticipated expiration: 2043-03-22
Also published as: CN116010228A

Abstract

The embodiment of the invention provides a time estimation method and a device for network security scanning, which relate to the technical field of network security, wherein the method comprises the following steps: estimating the scanning time by abstracting the task quantity and the task throughput in an intelligent estimation system by utilizing task information of a task queue and state information of a scanning module in a network security scanning system, and estimating the scanning time by extracting data features through a machine learning model; and performing time kneading processing on the plurality of estimated results to obtain a first time length of the network security scanning process. In addition, the estimated result obtained by combining multiple schemes is kneaded to obtain an exact time estimated value, so that the probability that a single scheme is influenced by accidental factors can be reduced, the accuracy of time estimation in the scanning process is further improved, and the technical problem of poor accuracy of time estimation in the scanning process in the prior art can be solved.

Description

Time estimation method and device for network security scanning

Technical Field

The present invention relates to the field of network security technologies, and in particular, to a method and apparatus for estimating time for network security scanning.

Background

In network security, network security scanning (hereinafter referred to as scanning) is a technology for detecting and extracting security information from objects through a computer network, and is a core technical method for evaluating network security of an information system. In a broad sense, scanning involves a number of aspects of information gathering, asset grooming, service detection, vulnerability detection, and the like. For example: the method comprises the steps of scanning a web site, firstly, scanning to obtain information such as how many pages of the site and a specific server operated by the site, then, further scanning to obtain more detailed technical information such as a technical framework used by the site and other network services started by the server according to the obtained information, and then, aiming at the specific technical information, carrying out corresponding vulnerability scanning to obtain vulnerability information, and finally, summarizing all the information to give out a result of a certain concern in network security.

Network security scanning encompasses other forms in addition to the above example forms, which are provided for illustration only: the scanning workload varies, and the workload of the whole scanning process is influenced by various factors such as the scale of a scanning target, the condition in the scanning process and the like.

In addition to uncertainty of the scanning task amount, the throughput of the scanning system for a certain target related task is difficult to determine, in a large-scale scanning system, the scanning system can be a distributed system formed by a plurality of network computing devices together, and the whole system has the capability of coping with different scanning tasks from self-allocation computing resources, so that the time required by the scanning system for processing the target of the same task amount is different at different times; meanwhile, whether other targets wait for processing or not on the system and the original busyness of the whole system also influence the throughput capacity of the system on the task quantity related to a specific target.

Therefore, it is difficult to estimate the time required for a scanning process, and the amount of work has uncertainty as to the network environment of the scanner and the scanned party during the scanning process. It can be seen that how to accurately predict the time required for a scanning process is an urgent issue to be resolved.

Disclosure of Invention

In view of the above, the embodiment of the invention provides a time estimation method for network security scanning, which aims to solve the technical problem of poor accuracy of scanning time estimation in the scanning process in the prior art. The method comprises the following steps: acquiring task information of a task queue and state information of a scanning module in a network security scanning system, wherein the network security scanning system is a system for executing a network security scanning process; according to the task information and the state information of the scanning module, a first estimated time interval during scanning of the target network security scanning process is estimated by abstracting the task quantity and the task throughput in an intelligent estimation system, and a second estimated time interval during scanning of the target network security scanning process is estimated by extracting data features through a machine learning model which is trained in advance; and performing time kneading processing on a plurality of estimated results including a first estimated time interval and a second estimated time interval to obtain a first time length of a target network security scanning process.

The embodiment of the invention also provides a time estimation device for network security scanning, which is used for solving the technical problem of poor accuracy of scanning time estimation in the scanning process in the prior art. The device comprises: the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring task information of a task queue and state information of a scanning module in a network security scanning system, and the network security scanning system is used for executing a network security scanning process; the time estimating module is used for estimating a first estimated time interval when the target network is scanned in the security scanning process by abstracting the task quantity and the task throughput in the intelligent estimating system according to the task information and the state information of the scanning module, and estimating a second estimated time interval when the target network is scanned in the security scanning process by extracting data features through a machine learning model which is trained in advance; and the time kneading module is used for carrying out time kneading processing on a plurality of estimated results comprising a first estimated time interval and a second estimated time interval to obtain a first time length of the target network security scanning process.

The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the random time estimation method facing to network security scanning when executing the computer program so as to solve the technical problem of poor accuracy of the estimation of the scanning time of the scanning process in the prior art.

The embodiment of the invention also provides a computer readable storage medium which stores a computer program for executing the random network security scanning-oriented time estimation method, so as to solve the technical problem of poor accuracy of scanning time estimation in the scanning process in the prior art.

Compared with the prior art, the beneficial effects that above-mentioned at least one technical scheme that this description embodiment adopted can reach include at least: the time estimation scheme for network security scanning is realized, and the problem that a user cannot obtain the residual time consumption required by a scanning process by using a complex network security scanning system in the past is solved; the method for combining the multi-time estimation model is designed, so that factors in multiple aspects of a scanning target side, a scanning system side and a historical data side can be taken into consideration, all factors affecting the scanning time of a complex scanning system are taken into consideration, and the accuracy of time estimation is improved; the time kneading and optimizing method is designed and realized, and the time estimation result is well optimized according to the actual running condition of the scanning process.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flowchart of a method for estimating time for network security scanning according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a time estimation scheme for network security scanning according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a relationship between a scanning system and a data acquisition function provided by an embodiment of the present invention;

FIG. 4 is a flowchart of an implementation process of an expert model quantitative estimation scheme provided by an embodiment of the present invention;

fig. 5 is a flowchart of a time kneading and optimizing scheme implementation process provided by an embodiment of the present invention;

FIG. 6 is a schematic diagram of a computer device according to an embodiment of the present invention;

fig. 7 is a block diagram illustrating a structure of a time estimation device for network security scanning according to an embodiment of the present invention.

Detailed Description

Embodiments of the present application are described in detail below with reference to the accompanying drawings.

Other advantages and effects of the present application will become apparent to those skilled in the art from the present disclosure, when the following description of the embodiments is taken in conjunction with the accompanying drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. The present application may be embodied or carried out in other specific embodiments, and the details of the present application may be modified or changed from various points of view and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

For the convenience of understanding the technical solutions of the present application, the following explains the technical terms related to the present application:

scanning consumers: in a scanning system, a program (or process) that is responsible for actually performing a scanning task, herein referred to as a scanning consumer.

Task amount: the invention refers to a quantized value of the work to be performed by the scanning system during the scanning process.

Throughput: the invention refers to the task amount which can be processed by the network security scanning system in the unit time of the current moment.

Network security scanning (hereinafter also referred to simply as "scanning") is a network security technique that detects objects over a computer network and extracts security information. The scanning process, namely a network security scanning system, is an object of time estimation for the process of executing network security scanning initiated by the appointed target. The estimated time of the scanning process can be compared with the estimated time of the physical examination of the human body to a certain extent, the whole physical examination workload of the human body is influenced by the health condition of the human body, whether further special examination is needed after the routine examination is carried out, and the final physical examination time is also influenced by the factors of how many people visit the hospital on the same day, the busyness of doctors and the like. Analogically, it is very difficult to predict the time for a particular web-safe scanning session, as is the time required to predict in advance for a person that he will have given his physical examination.

In actual scanning, where extremely fine scanning strategies are used for large targets, the entire scanning process will take hours, and even in the case of scanning systems with other targets, the system will take longer to complete the scan. It can be seen that providing an effective estimated time for a user of the scanning system greatly assists the user in scheduling his own operations and evaluating the operating state of the scanning system.

In the existing domestic and foreign patents and papers, the research on the time estimation of the network security scanning is in a blank state, the time estimation of the network security scanning is not involved, the engineering implementation is generally realized by artificial rough estimation, and the method aims at the particularly simple security scanning, wherein the simple method means that the task amount of the network security scanning is fixed, the estimation time is simple, and the time estimation has no reference to the technology described by the invention.

In order to overcome the above problems, in an embodiment of the present invention, a time estimation method for network security scanning is provided to accurately estimate time when a network security scanning system scans a target, as shown in fig. 1, the method includes:

Step S11, task information of a task queue and state information of a scanning module in a network security scanning system are acquired, wherein the network security scanning system is a system for executing a network security scanning process.

In order to solve the macroscopic problem (accurate prediction during scanning), the task amount of a scanning process needs to be dequantized, which can be specifically realized by the following ways:

1) Task information of a task queue in a network security scanning system is acquired: the task types (such as domain name task, sub domain name task, web task, service task) included in each network security scanning process (including target network security scanning process) and the total number of tasks, the number of completed tasks and the number of tasks to be completed under each task type;

2) Collecting state information of each network computing device (namely a scanning module) in the network security scanning system: the method comprises the steps of processing frequency of a processor, core number of the processor, core utilization rate of the processor, scanning efficiency parameters of each type of consumer process, total capacity of a storage medium, data interaction frequency of the storage medium, use amount of the storage medium, network uplink bandwidth and network downlink bandwidth, wherein each type of consumer process is used for processing one type of task, and the scanning efficiency parameters are used for representing busy degree of the corresponding type of task.

Step S12, according to the task information and the state information of the scanning module, a first estimated time interval when the target network security scanning process is scanned is estimated by abstracting the task quantity and the task throughput in the intelligent estimation system, and a second estimated time interval when the target network security scanning process is scanned is estimated by extracting data features through a machine learning model which is trained in advance. An intelligent estimation system herein is a system capable of performing a temporal estimation based on the task volume and the task throughput, such as a system that performs an estimation according to predefined rules provided by an expert (which may also be referred to as an expert system).

In one embodiment of step S12, since the estimated time is proportional to the task amount and inversely proportional to the throughput capability of the system, after the task amount of one scanning process is quantized, the throughput capability of the current scanning system for the task in a unit time needs to be quantized, so as to further perform time estimation:

1) Summing the task quantity of each subtype task in a plurality of subtype tasks (such as domain name tasks, sub-domain name tasks, web tasks and service tasks, and other tasks can be included according to the requirement) of the target network security scanning process according to the task information to obtain the task quantity of the target network security scanning process;

2) Summing task throughput of each class of consumer processes (such as consumer processes corresponding to domain name tasks, consumer processes corresponding to sub domain name tasks, consumer processes corresponding to web tasks and consumer processes corresponding to service tasks) in the target network security scanning process according to the state information of the scanning module to obtain task throughput of the target network security scanning process, wherein the task throughput of the consumer processes is jointly determined by factors such as scanning efficiency parameters, configured storage and the like;

3) And estimating a first estimated time interval when the target network security scanning process is scanned by using the task quantity and the task throughput of the target network security scanning process.

For example: and multiplying the ratio between the task quantity of the target network security scanning process and the task throughput of the target network security scanning process by the error lower limit value and the error upper limit value of the error coefficient (the error lower limit value and the error upper limit value of the error coefficient are obtained by analyzing historical data) respectively to obtain the time interval lower limit value and the time interval upper limit value of the first estimated time interval.

In another embodiment of step S12, before the second estimated time interval during scanning of the target network security scanning process is estimated by performing data feature extraction by the machine learning model trained in advance, in order to extract the task volume feature of the scanning process, the corresponding machine learning model may be trained in advance, so as to use the machine learning model to estimate the time:

1) The method comprises the steps of carrying out format adjustment on original data (data which are collected and used for carrying out model training) so as to meet the requirement of model training, for example, carrying out unified format arrangement on input data, wherein the method comprises the steps of realizing feature data extraction and label normalization in a model training stage, and carrying out format arrangement on the input data in a scanning time evaluation;

2) Dividing the original data subjected to format adjustment into a training set, a verification set and a test set, wherein most data are classified into the training set, and a small amount of data are classified into the verification set and the test set;

3) Training by using a training set, a verification set and a test set to obtain a machine learning model, wherein the training set is used for training parameters in the machine learning model and adjusting network weights so as to realize model fitting, the test set is used for evaluating the generalization capability of the machine learning model, and the verification set is used for adjusting super parameters in the machine learning model and evaluating the capability of the machine learning model.

Step S13, time kneading processing is carried out on a plurality of estimated results including a first estimated time interval and a second estimated time interval, and a first time length of a target network security scanning process is obtained.

A single estimated result may have a larger error, and in order to reduce the error, the final estimated time result may be obtained by kneading the time reference results given by the multiple submodels:

1) Acquiring weights allocated to a plurality of estimated results, e.g. available weights

Denoted as +.>

Individual prediction result->

The assigned weights, i, are 1 to p (p is the number of predicted results), with [ -degree>

]Indicate->

The estimated result is->

Representing the lower limit value of the interval, & lt & gt>

Representing an upper limit value of the section;

2) Multiplying the upper limit value of the time interval of each estimated result in the plurality of estimated results by corresponding weight, namely

Then accumulating to obtain a first summation value, multiplying the lower limit value of the time interval of each estimated result in the plurality of estimated results by corresponding weight, namely +.>

Then accumulating to obtain a second summation value;

3) And after the first summation value and the second summation value are added, dividing the added value by the sum of the weights of the plurality of estimated results to obtain the first time length of the target network security scanning process.

The process is an accurate pre-estimation process, in the actual process, as the task quantity and the like can change in real time, the next pre-estimation can be triggered once the task quantity and the like change, and the output result can be regulated according to the actual running condition of the scanning system, and the specific implementation process is as follows:

step S14 (the step is an optional step), continuously monitoring the received information of the network security scanning system, triggering a checking mechanism when the information is changed, running the time optimizing method, re-confirming whether the estimated time (such as the first time length) accords with the actual running condition, and if not, re-generating a new residual time estimated value (such as the second time length), and returning to the network security scanning system. The method comprises the following steps:

1) Continuously monitoring task information of a task queue and state information of a scanning module in a network security scanning system;

2) Under the condition that at least one of the monitored task information and the monitored state information is changed, estimating a third estimated time interval when the target network security scanning process is scanned by abstracting task quantity and task throughput in an intelligent estimation system according to the monitored task information and the monitored state information, and estimating a fourth estimated time interval when the target network security scanning process is scanned by extracting data features through a machine learning model;

3) And performing time kneading processing on a plurality of estimated results including a third estimated time interval and a fourth estimated time interval to obtain a second duration of the target network security scanning process.

The specific implementation steps of the three steps are similar to the corresponding steps, and are not repeated here.

In the technical scheme of the application, task information of a network security scanning system is firstly acquired, and state information of each scanning module of the scanning system is acquired; after the related information of the scanning system is obtained, the task quantity and the task throughput are abstracted based on an intelligent estimation system, a rough result (namely a first estimated time interval) during scanning is estimated, meanwhile, data characteristics of the information are extracted, and a rough result (namely a second estimated time interval) for time estimation is also output based on a machine learning model trained in advance; after further obtaining the two or even possible multiple estimated results, an exact time value (i.e. the first time length) is obtained through a time kneading algorithm and returned to the network security scanning system. In addition, the estimated result obtained by combining multiple schemes is subjected to the exact time estimated value obtained by kneading, so that the probability that a single scheme is influenced by accidental factors can be reduced, the accuracy of time estimation in the scanning process is further improved, and the technical problem of poor accuracy of time estimation in the scanning process in the prior art can be solved.

In addition, after an exact time value is obtained through a time kneading algorithm, information data obtained by the continuous monitoring and receiving network security scanning system can be changed, a checking mechanism is triggered, whether the estimated time accords with the actual running condition or not is reconfirmed through a time optimization method, a new residual time estimated value is generated, and the new residual time estimated value is returned to the network security scanning system.

As an alternative embodiment, the technical solution of the present application may be divided into the following functions (may also be referred to as functional modules or modules), and the relationship between the functional modules is shown in fig. 2:

data acquisition function: the method has the main functions of acquiring task information of the scanning system by monitoring the task queue of the scanning system, monitoring and acquiring parameters such as the use condition of hardware resources of each host computer operated by the distributed scanning system, the number of consumer processes of each scanning module started and the busy condition of the consumer, and pushing various acquired monitoring data to other functional modules.

Expert model quantitative estimation function: the main function is to abstract the task amount of the scanning process according to the data quantification provided by the scanning system through analysis summary of the developer of the network security scanning system

And quantifying the throughput of an abstract scanning system +.>

And calculating and outputting a predicted interval of scanning time according to the two abstracted parameters

，/>

](i.e. the first estimated time interval), i.e. the output calculation time possible minimum +.>

And the possible maximum value

。

Machine learning model estimation function: the main functions of the method are that a new method for constructing feature vectors is provided according to historically accumulated scanning data, the method comprises the steps of classifying the scanning recorded in the historical data into unified labels in a segmented mode, then carrying out model training on the processed data through SVN (which is totally called Support VectorMachine, the core idea is to find a boundary area so as to effectively distinguish classification information, the area is searched by minimizing the distance between a sample point and an area boundary so as to form a most suitable classification decision line), KNN (which is totally called K-Nearest Neighbors, namely K nearest neighbor classification method), training a model according to marked data, predicting new data points according to the model, predicting labels of the new data points, namely classification to which the data belong), and the like, and carrying out model training on the trained data by comparing machine learning classifier algorithms such as SVN (which is totally called Support VectorMachine, the classification to which the data belongs And (5) model, optimizing and screening an optimal model. The model can output a time period label according to the input scanning system information, and can be mapped back to a time interval by the label

，/>

](i.e., the second estimated time interval) and finally outputting the minimum value +.>

Maximum->

。

The expert model (i.e., the intelligent estimation system) quantization estimation function and the machine learning model estimation function described above may also be combined into a time estimation function or module.

Time kneading and adjusting function (also can be divided into time kneading function and time adjusting function): the main function is that the time interval of the multiple outputs of the two upstream models or other expansion schemes (schemes capable of being estimated during scanning) is a specific estimated time value

(e.g. first time length) by ∈>

Can get the->

Is an effective factor (++) obtained by data analysis>

Can understand +.>

Simplified expression of->

Is a positive integer>

Score of less than 1, ">

Computationally equal to->

Divided by all->

Sums, e.g. of->

Equal to->

Divided by all->

And sum), the factor also leaves a flexible adjustment means for the whole system, and also provides access points for other expansion models in the later stage of the system. After a certain scanning process, the user is given a specific estimated time value +. >

After that, the module can directly receive the monitoring data of the data acquisition function and continuously adjust the remaining time according to the completion condition of the key tasks in the task queue of the scanning system as a check point>

(e.g., the second duration) and communicates to the user to make corrections to the remaining scan time.

The technical solution of the present application may be deployed on a network computing device, and the following describes a software system thereof, i.e. a specific implementation of each function in fig. 2:

1) Data acquisition function

The main effect of this function is to obtain all the data needed for time estimation from the scanning system, the functional internals of which are related to the scanning system, see fig. 3. The interior of the device comprises three sub-functions: task data, computing resource data, consumer data.

This section introduces some background description of the scanning system in describing the method of implementation due to its close association with the network security scanning system.

1.1 Task data subfunction

The task refers to the distribution of the scanning system itself to an assignable unit of the scanning consumer execution within its system, and there are multiple categories of tasks. A task queue refers to a sequence of tasks waiting to be consumed. In the scanning process of executing scanning by a specific scanning target, the total number of tasks is initially 1, when the scanning process finds a new target, the scanning system generates a new task and adds the new task into a task queue of the task to wait for being consumed, and when a task is acquired by a scanning consumer, the task is removed from the task queue.

Based on the background, the task data sub-function adopts a strategy of monitoring variation to acquire the task data which is newly added and removed in the task queue, and classifies the acquired data. By continuously acquiring the newly added task information in the task queue, summarizing the task according to the specific scanning process of the task, wherein the task summarized under one process comprises a plurality of types, recording the total number, the completed number and the number to be completed according to the types, and finally merging the classified and counted data with the data obtained by other sub-functions of the data acquisition function and transmitting the merged and counted data to a model or a functional module which is required in the follow-up.

1.2 Computing resource data subfunction

The function adopts a strategy of timing acquisition, and acquires the calculation resources of all network calculation devices in a scanning system every several seconds, and acquires the processing frequency of a processor, the number of processor cores, the utilization rate of the processor cores, the total capacity of a storage medium, the data interaction frequency of the storage medium, the use amount of the storage medium and the uplink and downlink bandwidth data of a network. And combining the data obtained by the data subfunctions of all the computing devices with the data obtained by the other data subfunctions, and transmitting the data to a model or a functional module which is required subsequently.

1.3 Consumer data subfunction)

Consumers (or scanning consumer sub-functions) are processes of the scanning system for actually executing scanning tasks, the consumers are divided into a plurality of categories, and corresponding to the categories of the scanning tasks, one type of scanning task is responsible for carrying out specific network security scanning work by one type of scanning consumer sub-function. And in a specifically oriented scanning system, there is a self-adjusting mechanism for the number of consumers on each network computing device, and the number can vary.

Based on the background above, the consumer data sub-function employs a strategy of timed collection, collecting the number of consumers on each computing device every few seconds, and categorizing the consumers by different consumer types. While for evaluating the busyness of a certain type of consumer, the busyness of the consumer is determined by scanning the efficiency parameter for a type x task

To describe, the calculation method is as follows:

，

wherein, the liquid crystal display device comprises a liquid crystal display device,

is->

N tasks which are recently completed in the type task, wherein n is positively correlated with the total amount of consumers for the type task, and the value of n is +.>

Representing the run time taken by the consumer to execute the ith sample task, +.>

Representing the total time that the ith sample task is placed in the task queue until the task is completed.

The consumer data sub-function combines the total amount and scanning efficiency of various types of consumers collected and summarized from each computing device with data obtained by other data sub-functions and transmits the data to a model or a functional module which is needed subsequently.

2) Expert model quantitative estimation function

The expert model quantization estimation function includes three sub-functions: task amount quantization, throughput quantization, and calculation at scanning time. See fig. 4 for the flow relationships of the sub-functions:

step S41, receiving data obtained by a data acquisition function, and delivering the data to two quantization sub-modules for processing;

step S42, abstracting the task quantity by a method of task quantity quantization sub-function description;

step S43, abstracting the throughput by the method described in the throughput quantization sub-function;

step S44, the method described in the scanning time-consuming computing sub-function is used for computing the estimated time interval after acquiring the task quantity and throughput.

2.1 Task quantity quantization sub-function

Task data sub-function of task quantity quantization function interfacing data acquisition function, when receiving data related to one task

According to the content in the data, firstly confirming the scanning process of the task>

Determining task-specific scanning consumer subfunction >

(also simply referred to as consumer process or consumer, here equivalent to determining which consumer process specifically handles the task), determining the level of complexity of task execution +.>

Wherein->

、/>

、/>

Is determined by the specific configuration strategy of the scanning system.

Then according to

Dividing the task into a scanning process of a time to be estimated, wherein the same task is +.>

The task of the process is again defined by

Is classified as->

Task amount of->

Is a kind of medium. The task quantity M is specifically composed of four subtype task quantities, and the task quantities are respectively: task amount of domain name task->

Task amount of sub-domain name task>

Task amount of web task->

Task amount of service task +.>

. For a certain task quantum type, in +.>

For example, a->

Wherein->

The sum of the sub-functions of the scanning consumers representing the tasks belonging to the domain name (on the basis of the previous determination of the processes of the consumers for processing each task one by one, the process of all the tasks under the task can be determined, namely the set of the processes is D), the tasks executed by the scanning consumers in the set belong to the domain name tasks, and the calculation of the quantum types of the other tasks is the same.

Therefore, four kinds of task quantities can be abstracted from the complicated scanning system for subsequent calculation. Above mentioned

、

、/>

The mapping relation of the scanning system is different according to the specific implementation mode of the scanning system, and various implementation modes exist, and the mapping relation can be obtained only by carrying out condition analysis on the specific scanning system.

2.2 Throughput quantization sub-function

The throughput quantization sub-function interfaces with the consumer data sub-function of the data acquisition function, consumer related data has been sorted by type in the previous data acquisition function, and whenever a new batch of consumer data is acquired, the throughput quantization sub-function uses for a class of x consumers

Characterizing single class consumer throughput. But a single class of consumer throughput can only reflect the current processing power of a particular scanning task and not the overall throughput of the system. In order to better cope with various tasks, the task amount is classified in the task amount quantization sub-function, and various consumers are also required to be combined in the process of quantifying the throughput.

Corresponding to four types of tasks in the task volume, all consumers are also categorized as follows from the specific implementation of the scanning systemOne of four classes: and performing corresponding v-type according to the d-type corresponding to the domain name task, the s-type corresponding to the subdomain name task, the w-type corresponding to the web task and the service task. For a certain class of throughput, taking class d as an example, class d task throughput

Wherein->

For a consumer collection belonging to class d, +.>

The scale coefficient of the x-class task in the d-class task set is confirmed by statistics of historical data of the scanning system. Other types of throughput computation methods are the same.

Four classes of throughput can thus be quantized out for subsequent computation. The above classification can also be adjusted according to the specific scanning system conditions.

2.3 Scanning time-consuming computing sub-function

The sub-function for scanning receives the data output of the task quantum function and the throughput quantum function, and with the above-mentioned work, the estimated time required by a scanning process can be roughly calculated, and the estimated time is not expected to be obtained directly, and only the upper and lower limits of the time are required to be output to obtain a time interval.

Estimating time

Can pass->

Calculated, wherein->

The values are taken among the four types d, s, w, v,

representing->

Task amount of class task->

Representing->

Throughput of the class task is determined by,ktaking as error coefficient

Two values are taken into the calculation, resulting in +.>

、/>

And outputting. />

Obtained by analysis of historical data such that +.>

、/>

For the practical use of the scanning procedure +. >

Can meet->

The probability of (2) is greater than 95% as a standard.

The expert model input, output and calculation methods are described in general terms.

3) Machine learning model estimation function:

based on long-term data accumulation of a scanning system, model training is performed by using a machine learning algorithm, and the model training is mainly divided into two sub-functions according to a functional structure: data preprocessing sub-function, time assessment model sub-function.

3.1 Data preprocessing sub-function

The function performs format arrangement on input data uniformly, wherein the format arrangement comprises two aspects: feature data extraction and label normalization are realized in the model training stage, and input data is subjected to format arrangement during scanning time evaluation.

In the model training stage, relevant data in an accumulated scanning system database is extracted by taking a scanning process as a unit, various configuration type data related to the configuration of a scanning system and record type data generated in the early stage of scanning are removed, identification type data and some irrelevant result type data which are butted for other systems are removed, and data which are inconvenient to be used as model training input, such as a multi-stage format in the data, are subjected to format adjustment for subsequent model training. And the recorded scanning time data are normalized and arranged, for example, the time data are upwards evidence obtained by uniformly taking ten seconds as a unit, so that model training is more reasonable.

The sub-function receives the incoming data during the scan time estimation after model training, and formats the data to be consistent with the data training phase format, fills in the vacant fields, and communicates the data to the time estimation model sub-function.

3.2 Time estimation model sub-function

The main part of the function is trained via a machine learning algorithm, and finally a time segment is output.

In the training process, the data are divided into three data of a training set, a verification set and a test set for model training and test effects, and the generated and compared model comprises a plurality of gradient lifting type algorithm models, a random forest type model, a K adjacent type model and other machine learning models. After a large amount of data which are arranged in a scanning system database are trained and subjected to algorithm tuning, and because the label value of the model is time, the optimal model is selected under the standard that the accuracy is as high as possible and the accumulated error of the whole test set is as small as possible when erroneous judgment occurs mainly through analysis of the accuracy and the accumulated error of the test set by the model, and in the example, the model which is finally trained based on the weight and the algorithm is finally adopted through comparison, so that a better effect is achieved.

After model training is completed, a main body part of a time estimation model sub-function is obtained, data obtained by data preprocessing is received and input into the model, and the data label is subjected to a normalization process taking ten seconds as a whole to be rounded up, so that the time label output by the model is obtained by taking ten seconds as a unit, and ten seconds of expansion are respectively carried out on the upper side and the lower side of the label output by the model, so that an output result of the whole sub-function can be obtained.

4) Time kneading and optimizing functions:

the function is responsible for receiving data of a plurality of upstream functions, finally kneading out a specific estimated time of the scanning process, checking key nodes of the scanning process according to the specific progress of the scanning process, triggering a time optimization mechanism, and updating a residual time estimated result value. The method is divided into three sub-functions according to functions and structures: a time interval kneading sub-function, a check point confirmation sub-function, and a time tuning sub-function. See fig. 5 for relationships between the sub-functions.

Step S51, a time interval of the output of an upstream model is obtained;

step S52, generating a specific time estimation result through a time interval kneading sub-function;

step S53, monitoring and triggering a time tuning mechanism by the method described in the check point confirmation sub-module;

Step S54, re-tuning by the method described in the time tuning sub-function to generate a specific time estimation result;

step S55, outputting the time estimation result to the user.

4.1 Time interval kneading sub-function

The time interval kneading sub-function receives two time intervals output by the expert model quantitative estimation function and the machine learning model estimation function, kneads a specific estimated time result, and can knead more time intervals at the same time, thereby providing the boundary expandable capability for the whole system.

Specifically, a certain time interval is recorded

The time interval of [ ]>

](as described above [>

，/>

]、[/>

，/>

]) And attach a positive integer trusted weight value to it>

，/>

Obtained by analysis of the data and appropriately used according to the actual situation>

Allocating each time interval pair to be final->

The influence of (2) is then +.>

Can be expressed as +.>

. And transmitting T as a final result to the user and to the checkpoint confirmation sub-function, as a time estimate of the scanning process is recorded.

4.2 Check point validation sub-function

And a check point confirmation sub-function for receiving the data of the data acquisition function, recording task data of the scanning process in progress, the computing resource status of the whole scanning system and the scanning consumer, and simultaneously, a time interval kneading sub-function for recording and maintaining the time estimation result of the process.

When the following checkpoint conditions are monitored: (1) in task data of the scanning process, a task is found to be changed from an operation state to a completion state; (2) the time of the scanning process is already operated to 25%, 50% and 75% of the proportion value of the initial judgment result; (3) computing resource fluctuations exceeding 20% compared to previous recorded values; (4) the number of consumer changes by more than 20% for a certain class of scans. The data relating to the scanning process and the whole scanning system is passed to the time tuning sub-function.

4.3 Time tuning sub-function

And the time tuning sub-function is used for receiving the related data of the check point confirmation sub-function, tuning to obtain a new time estimation result and outputting the new time estimation result to the user and the check point confirmation sub-function.

Specifically, when task data change is received, that is, when the situation (1) above is described, only the proportion of the number of completed tasks to the total tasks needs to be changed, the proportion of the running time and the recorded time estimation result is checked correspondingly, when the difference between the two proportions is greater than 10%, the time estimation result is adjusted so that the time proportion is consistent with the task proportion, and a new time estimation result is output. If the conditions (2), (3), (4) and the like are received, the whole time estimation method needs to be restarted, and the scanning time estimation result is regenerated from the expert model quantitative estimation function and the machine learning model estimation function and output.

It should be noted that, in the technical scheme of the application, the multi-model structure can be expanded and integrated into more models; more check point conditions can be modified or set, and the time tuning method can be correspondingly changed; the specific data naming of the data acquisition may be adjusted.

By adopting the technical scheme, the time estimation scheme and the sight focus positioning scheme for network security scanning are realized, and the problems that a user cannot obtain the residual time consumption required by a scanning process by using a complex network security scanning system in the past are solved; the method for combining the multi-time estimation model is designed and realized, and factors in multiple aspects of a scanning target side, a scanning system side and a historical data side can be taken into consideration, so that all factors affecting the scanning time of a complex scanning system are taken into consideration; the time kneading and optimizing method is designed and realized, and the time estimation result is well optimized according to the actual running condition of the scanning process.

In this embodiment, a computer device is provided, as shown in fig. 6, including a memory 601, a processor 602, and a computer program stored in the memory and capable of running on the processor, where the processor implements any of the above-mentioned time estimation methods for network security scanning when executing the computer program.

In particular, the computer device may be a computer terminal, a server or similar computing means.

In this embodiment, a computer readable storage medium is provided, where the computer readable storage medium stores a computer program for executing any of the above-described time estimation methods for network security scanning.

In particular, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable storage media, as defined herein, does not include transitory computer-readable media (transshipment) such as modulated data signals and carrier waves.

Based on the same inventive concept, the embodiment of the invention also provides a time estimation device for network security scanning, such as the following embodiment. Because the principle of solving the problem of the time estimation device facing the network security scanning is similar to that of the time estimation method facing the network security scanning, the implementation of the time estimation device facing the network security scanning can be referred to the implementation of the time estimation method facing the network security scanning, and the repetition is omitted. As used below, the term "unit" or "module" may be a combination of software and/or hardware that implements the intended function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

Fig. 7 is a block diagram of a time estimation device for network security scan according to an embodiment of the present invention, as shown in fig. 7, including: the data acquisition module 71, the time estimation module 72, and the time kneading module 73 are described below.

A data acquisition module 71, configured to acquire task information of a task queue in a network security scanning system and status information of a scanning module, where the network security scanning system is a system for executing a network security scanning process;

The time estimation module 72 is configured to estimate a first estimated time interval during scanning in the target network security scanning process by abstracting a task amount and a task throughput in the intelligent estimation system according to the task information and the state information of the scanning module, and estimate a second estimated time interval during scanning in the target network security scanning process by extracting data features from a machine learning model trained in advance;

the time kneading module 73 is configured to perform time kneading processing on a plurality of estimated results including the first estimated time interval and the second estimated time interval, so as to obtain a first time length of the target network security scanning process.

In the above embodiment, the apparatus of the present application may further include: the time adjusting and optimizing module is used for: performing time kneading processing on a plurality of estimated results including a first estimated time interval and a second estimated time interval to obtain a first time length of a target network security scanning process, and continuously monitoring task information of a task queue and state information of a scanning module in a network security scanning system; under the condition that at least one of the monitored task information and the monitored state information is changed, estimating a third estimated time interval when the target network security scanning process is scanned by abstracting task quantity and task throughput in an intelligent estimation system according to the monitored task information and the monitored state information, and estimating a fourth estimated time interval when the target network security scanning process is scanned by extracting data features through a machine learning model; and performing time kneading processing on a plurality of estimated results including a third estimated time interval and a fourth estimated time interval to obtain a second duration of the target network security scanning process.

In the above embodiment, the data acquisition module is further configured to: acquiring task types and total number of tasks under each task type, number of completed tasks and number of tasks to be completed, which are included in each network security scanning process of the network security scanning system; the method comprises the steps of collecting the processing frequency of a processor, the number of processor cores, the utilization rate of the processor cores, the scanning efficiency parameters of each type of consumer process, the total capacity of a storage medium, the data interaction frequency of the storage medium, the use amount of the storage medium, the network uplink bandwidth and the network downlink bandwidth of each network computing device in a network security scanning system, wherein the network computing device is a scanning module, each type of consumer process is used for processing one type of task, and the scanning efficiency parameters are used for representing the busy degree of the corresponding type of task.

In the above embodiment, the time estimation module is further configured to: summing the task quantity of each subtype task in a plurality of subtype tasks of the target network security scanning process according to the task information to obtain the task quantity of the target network security scanning process, wherein the plurality of subtype tasks comprise domain name tasks, sub-domain name tasks, web tasks and service tasks; summing task throughput of each class of consumer processes in the multi-class consumer processes of the target network security scanning process according to the state information of the scanning module to obtain task throughput of the target network security scanning process, wherein the multi-class consumer processes comprise consumer processes corresponding to domain name tasks, consumer processes corresponding to sub domain name tasks, consumer processes corresponding to web tasks and consumer processes corresponding to service tasks; and estimating a first estimated time interval when the target network security scanning process is scanned by using the task quantity and the task throughput of the target network security scanning process.

Optionally, the time estimation module is further configured to: acquiring a ratio between the task quantity of the target network security scanning process and the task throughput of the target network security scanning process; and multiplying the ratio by the error lower limit value and the error upper limit value of the error coefficient respectively to obtain a time interval lower limit value and a time interval upper limit value of the first estimated time interval, wherein the error lower limit value and the error upper limit value of the error coefficient are obtained by analyzing historical data.

In the above embodiment, the time estimation module is further configured to: before a second estimated time interval in scanning of a target network security scanning process is estimated by extracting data features through a machine learning model which is trained in advance, carrying out format adjustment on original data so as to meet the requirement of model training, wherein the original data are collected data used for carrying out model training; dividing the original data subjected to format adjustment into a training set, a verification set and a test set, wherein the training set is used for training parameters in a machine learning model and adjusting network weights so as to realize model fitting, the test set is used for evaluating the generalization capability of the machine learning model, and the verification set is used for adjusting super parameters in the machine learning model and evaluating the capability of the machine learning model; and training by using the training set, the verification set and the test set to obtain a machine learning model.

In the above embodiment, the time kneading module is further configured to: acquiring weights distributed for a plurality of estimated results; multiplying the upper limit value of the time interval of each estimated result in the plurality of estimated results by a corresponding weight, accumulating to obtain a first summation value, multiplying the lower limit value of the time interval of each estimated result in the plurality of estimated results by a corresponding weight, and accumulating to obtain a second summation value; and after the first summation value and the second summation value are added, dividing the added value by the sum of the weights of the plurality of estimated results to obtain the first time length of the target network security scanning process.

It will be apparent to those skilled in the art that the modules or steps of the embodiments of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the invention are not limited to any specific combination of hardware and software.

The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, and various modifications and variations can be made to the embodiments of the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. The time estimation method for network security scanning is characterized by comprising the following steps:

acquiring task information of a task queue and state information of a scanning module in a network security scanning system, wherein the network security scanning system is a system for executing a network security scanning process;

according to the task information and the state information of the scanning module, a first estimated time interval when the target network security scanning process is scanned is estimated by abstracting the task quantity and the task throughput in an intelligent estimation system, and a second estimated time interval when the target network security scanning process is scanned is estimated by extracting data features through a machine learning model which is trained in advance;

performing time kneading processing on a plurality of estimated results including the first estimated time interval and the second estimated time interval to obtain a first time length of the target network security scanning process;

the acquiring task information of a task queue and state information of a scanning module in a network security scanning system comprises the following steps:

acquiring task types and total number of tasks under each task type, number of completed tasks and number of tasks to be completed, which are included in each network security scanning process of the network security scanning system;

The method comprises the steps of collecting the processing frequency of a processor, the number of processor cores, the utilization rate of the processor cores, the scanning efficiency parameters of each type of consumer process, the total capacity of a storage medium, the data interaction frequency of the storage medium, the use amount of the storage medium, the network uplink bandwidth and the network downlink bandwidth of each network computing device in a network security scanning system, wherein the network computing device is a scanning module, each type of consumer process is used for processing one type of task, and the scanning efficiency parameters are used for representing the busy degree of the corresponding type of task.

2. The time estimation method according to claim 1, wherein after time-kneading the plurality of estimated results including the first estimated time interval and the second estimated time interval to obtain the first time length of the target network security scan process, the method further comprises:

continuously monitoring task information of a task queue and state information of a scanning module in the network security scanning system;

under the condition that at least one of the monitored task information and the monitored state information is changed, estimating a third estimated time interval when the target network security scanning process is scanned by abstracting task quantity and task throughput in the intelligent estimation system according to the monitored task information and the monitored state information, and estimating a fourth estimated time interval when the target network security scanning process is scanned by extracting data features through the machine learning model;

And performing time kneading processing on a plurality of estimated results including the third estimated time interval and the fourth estimated time interval to obtain a second duration of the target network security scanning process.

3. The method of claim 1, wherein estimating the first estimated time interval for scanning the target network security scan process by abstracting the task amount and the task throughput in the intelligent estimation system based on the task information and the state information of the scan module comprises:

summing the task quantity of each subtype task in a plurality of subtype tasks of the target network security scanning process according to the task information to obtain the task quantity of the target network security scanning process, wherein the plurality of subtype tasks comprise domain name tasks, sub domain name tasks, web tasks and service tasks;

summing task throughput of each type of consumer process in the multi-type consumer process of the target network security scanning process according to the state information of the scanning module to obtain task throughput of the target network security scanning process, wherein the multi-type consumer process comprises a consumer process corresponding to a domain name task, a consumer process corresponding to a sub-domain name task, a consumer process corresponding to a web task and a consumer process corresponding to a service task;

And estimating the first estimated time interval of the scanning time of the target network security scanning process by using the task quantity and the task throughput of the target network security scanning process.

4. The method of time estimation according to claim 3, wherein estimating the first estimated time interval of the scan time of the target network security scan process using the task amount and the task throughput of the target network security scan process includes:

acquiring a ratio between the task amount of the target network security scanning process and the task throughput of the target network security scanning process;

and multiplying the ratio by the error lower limit value and the error upper limit value of the error coefficient respectively to obtain the time interval lower limit value and the time interval upper limit value of the first estimated time interval, wherein the error lower limit value and the error upper limit value of the error coefficient are obtained by analyzing historical data.

5. The method of any one of claims 1 to 4, wherein prior to estimating a second estimated time interval for scanning of the target network security scan process by data feature extraction using a pre-trained machine learning model, the method further comprises:

Carrying out format adjustment on the original data so as to meet the requirement of model training, wherein the original data is collected data for carrying out model training;

dividing the original data subjected to format adjustment into a training set, a verification set and a test set, wherein the training set is used for training parameters in the machine learning model and adjusting network weights so as to realize model fitting, the test set is used for evaluating generalization capability of the machine learning model, and the verification set is used for adjusting super parameters in the machine learning model and evaluating the capability of the machine learning model;

and training by using the training set, the verification set and the test set to obtain the machine learning model.

6. The method for estimating time according to any one of claims 1 to 4, wherein said time-kneading the plurality of estimated results including the first estimated time interval and the second estimated time interval to obtain the first time length of the target network security scan process includes:

acquiring weights distributed for the plurality of estimated results;

multiplying the upper limit value of the time interval of each estimated result in the plurality of estimated results by a corresponding weight, accumulating to obtain a first summation value, multiplying the lower limit value of the time interval of each estimated result in the plurality of estimated results by a corresponding weight, and accumulating to obtain a second summation value;

And after the first summation value and the second summation value are added, dividing the sum by the weight sum of the plurality of estimated results to obtain the first time length of the target network security scanning process.

7. A time estimation device for network security scanning, comprising:

the system comprises a data acquisition module, a data processing module and a data processing module, wherein the data acquisition module is used for acquiring task information of a task queue and state information of a scanning module in a network security scanning system, and the network security scanning system is used for executing a network security scanning process;

the time estimation module is used for estimating a first estimated time interval when the target network security scanning process is scanned by abstracting the task quantity and the task throughput in the intelligent estimation system according to the task information and the state information of the scanning module, and estimating a second estimated time interval when the target network security scanning process is scanned by extracting data features through a machine learning model which is trained in advance;

the time kneading module is used for carrying out time kneading processing on a plurality of estimated results comprising the first estimated time interval and the second estimated time interval to obtain a first time length of the target network security scanning process;

The data acquisition module is further configured to: acquiring task types and total number of tasks under each task type, number of completed tasks and number of tasks to be completed, which are included in each network security scanning process of the network security scanning system; the method comprises the steps of collecting the processing frequency of a processor, the number of processor cores, the utilization rate of the processor cores, the scanning efficiency parameters of each type of consumer process, the total capacity of a storage medium, the data interaction frequency of the storage medium, the use amount of the storage medium, the network uplink bandwidth and the network downlink bandwidth of each network computing device in a network security scanning system, wherein the network computing device is a scanning module, each type of consumer process is used for processing one type of task, and the scanning efficiency parameters are used for representing the busy degree of the corresponding type of task.

8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the network security scanning oriented time estimation method according to any of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for performing the network security scan-oriented time estimation method according to any one of claims 1 to 6.