CN116545737A

CN116545737A - Flow agent method, apparatus, computer device and storage medium

Info

Publication number: CN116545737A
Application number: CN202310626167.8A
Authority: CN
Inventors: 李曜晟; 潘琰; 吴悠; 何林飞
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-05-30
Filing date: 2023-05-30
Publication date: 2023-08-04

Abstract

The present application relates to a flow agent method, apparatus, computer device, storage medium and computer program product. Relates to the field of data processing, and can be used in the field of financial science and technology or other related fields. The method comprises the following steps: receiving a plurality of task information sent by a vulnerability mining platform, and sequencing the plurality of task information in a task processing queue based on the priority of the task information; acquiring the state of each proxy node, and distributing target task information in a task processing queue to target proxy nodes under the condition that the idle state of the target proxy nodes exists; the target task information is used for indicating the target proxy node to determine target flow request data; and constructing issuing information based on the determined target flow request data, and sending the issuing information to the vulnerability discovery platform. By adopting the method, the pressure of the proxy node can be reduced.

Description

Flow agent method, apparatus, computer device and storage medium

Technical Field

The present application relates to the field of data processing technology, and in particular, to a flow agent method, an apparatus, a computer device, a storage medium, and a computer program product.

Background

The current flow agent system can collect network flow data and filter the network flow data to obtain target flow data. Then, the flow agent system sends the target flow data to the vulnerability discovery platform. After the target flow data is acquired by the vulnerability mining platform, vulnerability analysis and vulnerability processing are carried out on the target flow data so as to improve the safety test efficiency.

The current flow agent method is that a first router in the flow agent system performs flow agent and records the flow data of the agent. The first router then mirrors the traffic data to the second router. And after receiving the flow data, the second router sends the flow data to a vulnerability-mining platform, and the vulnerability-mining platform filters the data to obtain target flow data.

However, in the current flow agent method, under the condition of excessive flow data or excessive concurrency of flow requests, a large pressure is caused on a router for performing flow agent, so that the agent router is down.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a flow agent method, apparatus, computer device, computer readable storage medium, and computer program product that address the above-described issues.

In a first aspect, the present application provides a flow agent method. The method comprises the following steps:

receiving a plurality of task information sent by a vulnerability mining platform, and sequencing the plurality of task information in a task processing queue based on the priority of the task information;

acquiring the state of each proxy node, and distributing target task information in the task processing queue to target proxy nodes under the condition that the idle state of the target proxy nodes exists; the target task information is used for indicating the target proxy node to determine target flow request data;

and constructing issuing information based on the determined target flow request data, and sending the issuing information to the vulnerability discovery platform.

In one embodiment, the sorting the task information in the task processing queue based on the priorities of the task information includes:

storing a plurality of task information into a preset initial task processing queue;

and sequentially sequencing the task information in the initial task processing queue based on the order of the priority of the preset task information from high to low to obtain a task processing queue.

In one embodiment, the distributing the target task information in the task processing queue to the target proxy node in the case that there is an idle target proxy node includes:

under the condition that an idle target agent node exists, determining first task information in the task processing queue as target task information;

transmitting the target task information to the target proxy node;

and establishing a corresponding relation between the address of the target agent node and the target task information.

In one embodiment, the constructing the issuing information based on the determined target traffic request data and sending the issuing information to the vulnerability discovery platform includes:

marking the target flow request data to obtain the label of the target flow data;

packaging labels corresponding to the target flow request data to obtain downlink information;

and storing the descending information into a preset task queue to be scanned, and sending the descending information to the vulnerability mining platform according to the sequence of the task queue to be scanned.

In one embodiment, before the step of packaging the label corresponding to the target flow request data and obtaining the downlink information, the method further includes:

Receiving task information to be ended sent by the vulnerability mining platform, and storing the task information to be ended to an ending task queue;

and reading target task information to be ended in the task ending queue, releasing the corresponding relation between the target task information and the address of the target proxy node, and updating the processing state of the target task information in the task processing queue based on the target task information to be ended.

In one embodiment, a flow agent system is provided, the system comprising:

the distribution server is used for receiving a plurality of task information sent by the vulnerability discovery platform, and sequencing the plurality of task information in a task processing queue based on the priority of the task information; acquiring the state of each proxy node, and distributing target task information in the task processing queue to target proxy nodes under the condition that the idle state of the target proxy nodes exists; the target task information is used for indicating the target proxy node to determine target flow request data; constructing issuing information based on the determined target flow request data, and sending the issuing information to the vulnerability discovery platform;

And the proxy server is used for receiving the target task information and determining target flow request data based on the target task information.

In one embodiment, the proxy server is specifically configured to obtain a plurality of flow request data;

acquiring a plurality of flow request data;

screening the plurality of flow request data based on the target task information to obtain initial flow request data;

and filtering the initial flow request data according to a preset matching rule to obtain target flow request data.

In a second aspect, the present application also provides a flow agent apparatus. The device comprises:

the sequencing module is used for receiving a plurality of task information sent by the vulnerability discovery platform, and sequencing the plurality of task information in a task processing queue based on the priority of the task information;

the distribution module is used for acquiring the state of each proxy node and distributing the target task information in the task processing queue to the target proxy node under the condition that the idle target proxy node exists; the target task information is used for indicating the target proxy node to determine target flow request data;

And the sending module is used for constructing issuing information based on the determined target flow request data and sending the issuing information to the vulnerability discovery platform.

In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:

In a fourth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program which, when executed by a processor, implements the steps of:

The flow agent method, the flow agent device, the computer equipment, the storage medium and the computer program product are used for receiving a plurality of task information sent by a vulnerability discovery platform and sequencing the plurality of task information in a task processing queue based on the priority of the task information; acquiring the state of each proxy node, and distributing target task information in the task processing queue to target proxy nodes under the condition that the idle state of the target proxy nodes exists; the target task information is used for indicating the target proxy node to determine target flow request data; and constructing issuing information based on the determined target flow request data, and sending the issuing information to the vulnerability discovery platform. By adopting the method, the task information sent by the vulnerability mining platform is sequenced in the task processing queue, and the target task information in the task processing queue is sent to the proxy node under the condition that the proxy node is in an idle state, the proxy node carries out flow proxy based on the target task information, the concurrency of the proxy node is controlled, and the data volume of target flow request data of the proxy node proxy is reduced. Further, the pressure of the proxy node is reduced.

Drawings

FIG. 1 is an application environment diagram of a traffic agent method in one embodiment;

FIG. 2 is a flow diagram of a method of traffic brokering in one embodiment;

FIG. 3 is a flow diagram of a process for determining a task processing queue in one embodiment;

FIG. 4 is a flow diagram of the steps for distributing target task information in one embodiment;

FIG. 5 is a schematic diagram of an execution flow of task distribution in one embodiment;

FIG. 6 is a flow chart illustrating the steps of sending a message in one embodiment;

FIG. 7 is a flowchart illustrating a step of releasing the correspondence in one embodiment;

FIG. 8 is a flow diagram illustrating the execution of a flow packing module in one embodiment;

FIG. 9 is a flow chart illustrating the steps for determining target flow request data in one embodiment;

FIG. 10 is a flow diagram of the execution of a traffic agent node in one embodiment;

FIG. 11 is a diagram of a pre-deployment headquarter data center network topology in one embodiment;

FIG. 12 is a diagram of a post-deployment headquarter data center network topology in one embodiment;

FIG. 13 is a diagram of a network topology model of a headquarter data center in one embodiment;

FIG. 14 is a diagram of a post-deployment, split-row data center network topology in one embodiment;

FIG. 15 is a diagram of a network topology model of a split data center in one embodiment;

FIG. 16 is a flow chart illustrating the execution of a flow agent system in another embodiment;

FIG. 17 is a block diagram of a flow agent apparatus in one embodiment;

fig. 18 is an internal structural view of a computer device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

The flow agent method provided in the embodiment of the present application may be applied to the flow agent system 100 shown in fig. 1. The flow agent system 100 includes a distribution server 110, a proxy server 120, and a database server 130. The distribution server 110 is communicatively connected to the proxy server 120 and the database server 130, respectively. The proxy server 120 may be implemented as a stand-alone server or as a proxy server cluster composed of multiple servers. The proxy node is a proxy server 120. Proxy server 120 is communicatively coupled to data server 130. The data storage module is integrated on the database server 130, and is used for storing processing data and processing results in the processing procedure of the distribution server 110 and the proxy server 120.

In one embodiment, as shown in fig. 2, a flow agent method is provided, which is described by taking as an example that the method is applied to the distribution server 110 (hereinafter omitted from numbering as a distribution server) in fig. 1, and includes the following steps:

step 202, receiving a plurality of task information sent by the vulnerability discovery platform, and sequencing the plurality of task information in a task processing queue based on the priority of the task information.

The distribution server is preset with a priority of task information and an initial task processing queue. The distribution server is in communication connection with the vulnerability discovery platform.

In implementation, before the distributing server receives the task information, the vulnerability discovery platform needs to determine a plurality of task information that needs to be flow proxied. The vulnerability discovery platform establishes communication connection with the distribution server and sends a plurality of task information to the distribution server. After the distribution server receives the task information sent by the vulnerability discovery platform, the distribution server stores the task information into a preset initial task processing queue. After the task information is stored, the distribution server sorts the plurality of task information in the initial task processing queue based on the priority of the preset task information, and a task processing queue is obtained.

Step 204, the state of each proxy node is obtained, and the target task information in the task processing queue is distributed to the target proxy node under the condition that the idle target proxy node exists.

The target task information is used for indicating the target proxy node to determine target flow request data.

In an implementation, a distribution server obtains the status of each proxy node. Based on the state of each proxy node, it is determined whether each proxy node of the plurality of proxy nodes is in an idle state. If the state of the proxy node is in an idle state, the distribution server determines the proxy node as a target proxy node. Then, the distribution server distributes the target task information in the task processing queue to the target proxy node. If the state of the agent node is not in the idle state, the distribution server reacquires the state of each agent node. Then, the distribution server executing the next round judges whether each of the plurality of proxy nodes is in an idle state until there is a target proxy node in the idle state.

And 206, constructing issuing information based on the determined target flow request data, and sending the issuing information to the vulnerability discovery platform.

In an implementation, after determining that the target flow request data is completed, the distribution server marks the target flow request data, thereby obtaining an identification of the target flow request data. Then, the distribution server constructs the delivery information based on the identification of the target traffic request data. Further, the distribution server transmits the outgoing message to the vulnerability discovery platform.

In the flow agent method, the task processing queues are used for sequencing the task information sent by the vulnerability mining platform, and the target task information is sent to the agent node when the agent node is in an idle state, so that the agent node carries out flow agent based on the target task information, the concurrency of the agent node is controlled, and the data volume of target flow request data of the agent node agent is reduced. Further, the pressure of the proxy node is reduced.

In one embodiment, as shown in fig. 3, the step 202 of sorting the plurality of task information in the task processing queue based on the priority of the task information includes:

step 302, storing a plurality of task information into a preset initial task processing queue.

In an implementation, an initial task processing queue is preset in the distribution server. After receiving the plurality of task information, the distribution server stores the plurality of task information into a preset initial task processing queue.

Step 304, based on the order of the priority of the preset task information from high to low, sequentially ordering the task information in the initial task processing queue to obtain a task processing queue.

In implementation, the distribution server is preset with priorities of task categories. The distribution server determines a task category of each task information. And then, the distribution server sequentially sorts the task information in the initial task processing queue according to the order of the priority from high to low based on the priority of the preset task class, so as to obtain a task processing queue.

In an alternative embodiment, the priorities of the task information preset in the distribution server are as shown in table 1 below:

TABLE 1

In table 1, the priorities are CS7, CS6, EF, AF4, AF3, AF2, AF1, BE in the order from high to low. Task categories (services) include protocol messages, host services, tandem services, video, voice, switching services, bulk, OA (Office Automation ), testing, and others. Each task class corresponds to a different priority. Specifically: the protocol messages may correspond to CS7 or CS6 priority based on different levels of the protocol. The absolute priority of the host traffic or the alien tasks (i.e., inter-host traffic) is EF priority. The voice service (namely telephone service) has high requirements on real-time performance and importance, and the corresponding priority is AF4. The video service is a video conference service and the like, and the corresponding priority is AF4. The priority corresponding to the switching service is AF3. The batch service is a production service with large data volume, and the corresponding priority is AF2.OA (Office Automation System, office automation) services include office services and related website video monitoring, etc., test services exist mainly between data centers and define bandwidths. The priority corresponding to the OA service and the test service is AF1. The priority corresponding to other services is BE. The distribution service determines a task category for each task information in the initial task processing queue. And then, the distribution server sequentially sorts the task information according to the order from high to low based on the priority of the preset task class, so as to obtain a task processing queue.

In this embodiment, by storing each task information in the initial task processing queue and sorting each task information according to the priority of the task information, sorting each task information in the task processing queue is completed, omission of task information is avoided, task information with higher priority can be processed in time, and timeliness of important task information processing is guaranteed.

In one embodiment, the specific process of distributing the target task information in the task processing queue to the target agent node in the presence of the target agent node in the idle state in step 204 includes:

step 402, determining first task information in a task processing queue as target task information in the case that there is a target agent node in an idle state.

In the implementation, when the idle target proxy node exists, the distributing server determines first task information in the task processing queue as target task information according to the order of the task processing queue.

Step 404, the target task information is sent to the target proxy node.

In implementations, the target proxy node is integrated in the proxy server. After determining the target task information, the distribution server transmits the target task information to the proxy server.

Step 406, establishing a corresponding relation between the address of the target agent node and the target task information.

In an implementation, after sending the target task information to the target proxy node, the distribution server obtains the address of the target proxy node. Then, the distribution server establishes a correspondence relationship between the address of the target agent node and the target task information.

In an alternative embodiment, a task distribution module is provided in the distribution server. The task distribution module includes a plurality of sub-modules as shown in table 2 below.

TABLE 2

Sub-module numbering	Sub-module name
		001-001	Dynamic proxy opening module
001-002	Task distribution creation module
		001-003	Task stop module
001-004	Access control module

The task distribution module comprises a dynamic agent opening module (with the number of 001-001), a task distribution creation module (with the number of 001-002), a task stopping module (with the number of 001-003) and an access stopping module (with the number of 001-004). The function of each sub-module is different. Specifically: the dynamic agent opening module is responsible for dynamically distributing agents to tasks according to the target task information and the states of the agent nodes. The task distribution creation module is responsible for creating proxy tasks in the task processing queues. The task stopping module is responsible for stopping the proxy task according to the received task information to be ended. The access control module is responsible for verification of the call authority.

In an alternative embodiment, the task distribution module task distribution execution flow, as shown in fig. 5, includes the following steps:

step 501, consuming a pre-proxy task queue.

In implementations, the pre-proxy task queue is a task processing queue. The distribution server determines task information in a pre-proxy task queue. Task information is target task information.

Step 502, judging the authority of the task information.

In an implementation, the distribution server determines whether the task information has rights.

Step 503, create pre-proxy tasks and update the database.

In implementations, the distribution server creates a pre-proxy task in the event that the task information has rights. The distribution server then sends the pre-proxy task to the database server. The database server stores the pre-proxy task after receiving the pre-proxy task.

In an alternative embodiment, the distribution server ends the processing of the task information in case the task information does not have rights.

Step 504, it is determined whether the proxy resources are sufficient.

In an implementation, a distribution server obtains the status of each proxy node. Then, the distribution server judges whether the state of each of the proxy nodes is an idle state.

Step 505, dynamically distributing agents.

In an implementation, task information is distributed to a target proxy node in the presence of the target proxy node in an idle state.

In this embodiment, when the proxy node is in an idle state, the target task information is sent to the proxy node, and the proxy node performs flow proxy based on the target task information, so that the concurrency of the proxy node is controlled, and the pressure of the proxy node is reduced.

In one embodiment, the specific process of step 206 includes:

and step 602, marking the target flow request data to obtain the label of the target flow data.

In implementations, the distribution server determines a unique identification of the target task information. Then, the distribution server marks the target flow request data according to the unique identification of the target task information, and obtains the label of the target flow data.

Step 604, packing the label corresponding to the target flow request data to obtain the downlink information.

In an implementation, the distribution server packages the labels of the target traffic request data after determining the labels of the target traffic request data to obtain the downlink information.

Step 606, storing the descending information in a preset task queue to be scanned, and sending the descending information to the vulnerability mining platform according to the sequence of the task queue to be scanned.

In an implementation, a task queue to be scanned is preset in the distribution server. The task queue to be scanned is used for storing the issuing information. And the distribution server stores the downlink information into a preset task queue to be scanned. And then, the distribution server sends the descending information to the vulnerability discovery platform through communication connection according to the sequence of the task queues to be scanned.

In this embodiment, by determining the label of the target flow request data and sending the label of the target flow request data to the vulnerability discovery platform, the amount of data transmitted is reduced, and the speed of sending the target flow request data is increased.

In one embodiment, before executing step 604, the task information to be ended sent by the vulnerability discovery platform needs to be received, and the agent node and the task processing queue are processed according to the task information to be ended.

As shown in fig. 7, the specific processing procedure of the flow agent method further includes, before executing step 604:

step 702, receiving task information to be ended sent by the vulnerability discovery platform, and storing the task information to be ended to an ending task queue.

In implementation, before the distributing server receives the task information to be ended, the vulnerability discovery platform needs to determine the task information to be ended that needs to end the flow agent. After the task information to be ended is determined, the vulnerability discovery platform sends the task information to be ended to the distribution server through communication connection. And the distribution server receives task information to be ended, which is sent by the vulnerability discovery platform. Then, the distributing server stores the task information to be ended into a preset ending task queue.

Step 704, reading target task information to be ended in the task ending queue, releasing the corresponding relation between the target task information and the address of the target proxy node, and updating the processing state of the target task information in the task processing queue based on the target task information to be ended.

The target to-be-ended information is target task information.

In implementation, the distributing server reads the target task information to be ended according to the order of the task ending queues. Then, the distribution server releases the correspondence between the target task information and the address of the target proxy node based on the target to-be-ended information. After the correspondence is released, the distribution server updates the processing state of the target task information in the task processing queue to an end state.

In an optional embodiment, a flow packing module is disposed in the distributing server, where the flow packing module is used to implement functions such as task state pushing, system state pushing, proxy flow packing, and task to be scanned issuing. Specifically, the execution flow of the flow packing module is shown in fig. 8, and includes the following steps:

step 801, query and push system status at regular time.

In implementation, the distribution server queries the state of the flow agent system according to a preset first query period. The distribution server then pushes the status of the flow agent system to the target user.

Step 802, query and push agent task status at regular time.

In implementation, the distributing server queries the state of each task information in the task processing queue according to a preset second query period. Then, the distribution server pushes the status of each task information to the target user.

In step 803, the proxy task queue to be ended is consumed.

In an implementation, the distribution server determines target to-be-ended task information in a to-be-ended proxy task queue.

Step 804, ending the proxy and releasing the occupied proxy resource.

In an implementation, the distribution server releases the correspondence between the target task information and the address of the target proxy node.

Step 805, the current task agent is ended and the unique task label is issued.

In an implementation, the distribution server updates the processing state of the target task information in the task processing queue based on the target to-be-ended information. The distribution server marks the target flow request data to obtain the label of the target flow data, and the label is the ID (Identity document, unique code) of the target task information.

Step 806, the task is issued to the task queue to be scanned.

In the implementation, the distributing server packages the label corresponding to the target flow request data to obtain the downlink information. And then, the distribution server stores the downlink information into a preset task queue to be scanned.

And, the flow packing module includes a plurality of sub-modules, as shown in table 3 below.

TABLE 3 Table 3

The task distribution module comprises a task state update module (with the number of 003-001), a system state update module (with the number of 003-002), a flow packing module (with the number of 003-003) and a task issuing module to be scanned (with the number of 003-004). The function of each sub-module is different. Specifically: the task state updating module queries and pushes the state of the task information at regular time. The system state updating module queries and pushes the flow agent system state at regular time. The flow packing module is responsible for consuming the task queue to be ended and packing the labels of all captured flows of the task to obtain the downlink information. The task issuing module to be scanned is responsible for issuing the packaged issuing information to a task queue waiting for scanning by the scanner.

In this embodiment, the task information to be ended is obtained, the corresponding relationship between the target task information and the target proxy node is released based on the task information to be ended, and the state of the target task information is updated, so that the state of the target proxy node becomes idle, and the subsequent proxy node is convenient to perform flow proxy.

In one embodiment, as shown in FIG. 1, a flow agent system 100 is provided, the flow agent system 100 comprising:

A distribution server 110, configured to receive a plurality of task information sent by the vulnerability discovery platform, and order the plurality of task information in a task processing queue based on priorities of the task information; acquiring the state of each proxy node, and distributing target task information in a task processing queue to target proxy nodes under the condition that the idle state of the target proxy nodes exists; the target task information is used for indicating the target proxy node to determine target flow request data; and constructing issuing information based on the determined target flow request data, and sending the issuing information to the vulnerability discovery platform.

In implementation, the specific processing procedure of the distribution server 110 is specifically described in the embodiments of the above-mentioned flow agent method, and the embodiments of the present application are not described herein in detail.

The proxy server 120 is configured to receive the target task information and determine target traffic request data based on the target task information.

In implementation, proxy server 120 receives target task information sent by distribution server 110. After obtaining the target task information, the proxy server 120 acquires the traffic request data. The proxy server 120 determines target traffic request data among the traffic request data based on the target task information.

In this embodiment, the distributing server orders the plurality of task information sent by the vulnerability discovery platform in the task processing queue, and sends the target task information in the task processing queue to the proxy node when the proxy node is in an idle state, and the proxy node performs flow proxy based on the target task information, controls concurrency of the proxy node, and reduces data volume of target flow request data of proxy node proxy. Further, the pressure of the proxy node is reduced.

In one embodiment, the specific process of determining the target traffic request data by the proxy server based on the target task information includes:

at step 902, a plurality of traffic request data is obtained.

In implementation, proxy server 120 receives a plurality of traffic requests and obtains a plurality of traffic request data corresponding to the plurality of traffic requests.

Step 904, screening the plurality of flow request data based on the target task information to obtain initial flow request data.

In implementation, proxy server 120 determines, for each of the plurality of traffic request data, whether the traffic request data is from the target task information. If the traffic request data is from the target task information, the proxy server 120 determines the traffic request data as initial traffic request data. If the traffic request data does not come from the target task information, proxy server 120 discards the traffic request data.

Step 906, filtering the initial flow request data according to a preset matching rule to obtain target flow request data.

In implementation, the matching rules are preset in the proxy server 120. The matching rule includes a regular expression and a keyword. The proxy server 120 filters the initial flow request data based on a preset matching rule to obtain target flow request data.

In an alternative embodiment, proxy 120 has configured therein a mittproxy (a hypertext transfer protocol proxy supporting secure socket protocols) traffic proxy node. The execution flow of the Mitmproxy flow agent node is shown in fig. 10, and comprises the following steps:

step 1002, a traffic request is received.

In implementation, proxy server 120 receives a plurality of traffic requests, resulting in a plurality of traffic request data.

Step 1004, determines whether from the current task agent target.

At step 1006, traffic statistics is performed and the database is updated.

In implementation, proxy server 120 counts the number of URLs (uniform resource location systems) contained in received traffic requests. The proxy server 120 then sends the number of URLs to the database server 130. Database server 130 stores the number of URLs.

At step 1008, the traffic is filtered and stored in a database.

In implementation, the proxy server 120 is pre-set with matching rules. The matching rule includes a regular expression and a keyword. The proxy server 120 filters the initial flow request data based on a preset matching rule to obtain target flow request data. The proxy server 120 then sends the target traffic request data to the database server 130. The database server 130 stores the target traffic request data.

In step 1010, traffic is forwarded.

In an implementation, the proxy server performs traffic forwarding based on the target traffic request data.

And, the Mitmproxy traffic agent node contains a plurality of sub-modules, as shown in Table 4 below.

TABLE 4 Table 4

The Mitmproxy flow agent node comprises a flow interception module (number 002-001), a flow filtering module (number 002-002), a flow statistics module (number 002-003) and a flow forwarding module (number 002-004). The function of each sub-module is different. Specifically: the traffic interception module is responsible for intercepting traffic and judging whether the traffic is from the current task target. The flow statistics module is responsible for counting the current captured flow quantity in real time and updating the database. The flow filtering module is responsible for filtering and integrating the required flow information and storing the information into the database. The flow forwarding module is responsible for transparent proxy to the flow and ensures normal access.

In the embodiment, the target flow request data is determined in the flow request data based on the target proxy task and the matching rule, so that the further filtering of special flow is realized, and the filtering capability of the proxy server is improved.

In an alternative embodiment, the flow agent system 110 further includes a database server 130, and the database server 130 has a data storage module integrated therein. The data storage module is mainly composed of a mongdb database (a database based on distributed file storage). The data storage module includes a plurality of sub-databases as shown in table 5 below.

TABLE 5

Sub-module numbering	Sub-module name	Description of the functionality
			005-001	Task information database	Storing user-added proxy task information
005-002	Agent information database	Storing proxy configuration information
			005-003	Capturing traffic database	Storing captured traffic data
005-004	Statistical analysis database	Storing traffic statistics

The data storage module mainly comprises a task information database (with the number of 005-001), an agent information database (with the number of 005-002), a captured flow database (with the number of 005-003) and a statistical analysis database (with the number of 005-004). The task information database is responsible for storing agent task information added by a user, and specifically comprises information such as a scanning target, a scanning state, a specific self-selection scanning plug-in, scanning time, timing execution and the like. The agent information database is responsible for storing information such as dynamically allocated agent addresses, current agent tasks, agent states, and the like. The captured traffic database is responsible for storing the captured task traffic data. The statistical analysis database is responsible for storing traffic statistics.

In an alternative embodiment, the user needs to deploy and configure the flow agent system prior to using the flow agent system. In deploying the flow agent system, the flow agent system needs to be deployed in the core backbone.

For example, if a banking system needs to deploy the flow agent system, it needs to be deployed in two places with the core backbone of the banking system: between a branch and a general line data center or between a branch and a net point. The headquarter data center network topology is shown in fig. 11 prior to deployment of the flow agent system. Wherein in fig. 11, DC represents a data center. If the flow agent system is deployed between the branch and the headquarter data center, the topology of the headquarter data center network after deployment is shown in fig. 12. The deployment location is located at the circle in fig. 12. Based on the network topology structure diagram of the headquarter data center, a network topology model diagram of the headquarter data center can be obtained, as shown in fig. 13. In fig. 13, the data centers that interact with the three data centers are the total line data centers, and the other data centers are the branch line data centers. If the flow agent system is deployed between the branch data center and the network point, the deployed network topology of the branch data center is shown in fig. 14. Based on the network topology structure diagram of the branch data center, a network topology model diagram in the branch data can be obtained, as shown in fig. 15. In fig. 15, a is a line data center, and R is a dot.

In addition, in deploying the flow agent system, the distribution server, database server, and proxy server in the flow agent system need to meet the operating environment requirements as shown in table 6 below.

TABLE 6

In configuring the flow agent system, the system of distributing servers is windows (Microsoft Windows, an operating system) or linux (an operating system), and the memory of the system needs to be greater than 8G (gigabytes), and the storage space needs to be greater than 100G (gigabytes). In addition, a running environment integrated with Python3 (a scripting language) is also required on the system of the distribution server. The database server system is windows or linux, and the memory of the system needs to be larger than 16G, and the memory space needs to be larger than 500G. In addition, the system of the database server also needs to have mongab integrated thereon. The proxy server system is windows or linux, and the memory of the system needs to be larger than 16G, and the memory space needs to be larger than 500G. In addition, the proxy server system also needs to integrate with the running environment of Python3 and the mitmvroxy proxy node.

After completing the deployment of the flow agent system, the user needs to configure the interface of the flow agent system. Specifically, the interfaces of the flow agent system include two types of interfaces, one being a user interface and the other being an agent interface. The user interface configuration process is to write the information such as url (uniform resource locator ) and token (identity token) into the configuration file, and double-click to run the proxy service. Then, the user configures a plurality of proxy interfaces including a proxy task creation interface, a proxy task stop interface, a proxy task deletion interface, a system status transmission interface, a proxy task status transmission interface, and a proxy result query transmission interface. The proxy task creation interface is shown in table 7 below:

TABLE 7

In table 7, the proxy task creation interface is connected with a POST/api/v1/task, a token is an identity token, and a Content-Type is a Content Type.

The proxy task stop interface is shown in table 8 below:

TABLE 8

In table 8, the connection of the agent task stop interface is POST/api/v1/task/stop, token is identity token, and Content-Type is Content Type.

The proxy task delete interface is shown in table 9 below:

TABLE 9

In table 9, the connection of the proxy task deletion interface is DELETE/api/v1/task, token is identity token, and Content-Type is Content Type.

The system status transmission interface is shown in the following table 10:

table 10

In table 10, the connection of the system status sending interface is POST/api/v1/save_scanner_status, token is identity token, and Content-Type is Content Type.

The proxy task state send interface is shown in table 11 below:

TABLE 11

In table 11, the proxy task state sending interface is connected to be POST/api/v1/save_task, token is an identity token, and Content-Type is a Content Type.

The proxy result query send interface is shown in table 12 below:

table 12

In table 12, the proxy result query sending interface is connected with POST/api/v1/save_vuln, token is an identity token, and Content-Type is a Content Type.

After completing the deployment of the flow agent system, the user initiates the flow agent system to conduct flow agent. The flow agent system adopts a distributed architecture based on a message queue, and is mainly divided into a pre-agent task queue (task processing queue), a pre-agent ending task queue (ending task queue), a task distribution module, a flow agent service node and a flow packing module. The flow agent system execution flow chart is shown in fig. 16, and includes the following steps:

step 1601, receiving task information sent by the vulnerability discovery platform, and storing the task information in a task processing queue.

In practice, the proxy server is pre-configured with an initial task processing queue. And the proxy server receives the plurality of task information sent by the vulnerability discovery platform and stores the plurality of task information into an initial task processing queue. Then, the proxy server sorts the plurality of task information in the initial task processing queue based on the priority of the task information, and a task processing queue is obtained.

In step 1602, pre-proxy task information is obtained from a pre-proxy task queue.

In an implementation, a task distribution module is provided in the distribution server. And a task distribution module in the distribution server determines the first task information in the task processing queue as pre-agent task information to obtain target task information.

Step 1603, judging the resource occupation condition of each proxy service node, selecting an idle proxy service address to bind task information and service according to the load balancing principle;

in an implementation, a task distribution module in a distribution server obtains the status of each proxy service node. In the case where there is a target agent node in an idle state, the distribution server distributes target task information to the target agent node. Then, the distribution server establishes a corresponding relation between the address of the target agent node and the target task information, and binding of the target agent node and the target task information is completed.

Step 1604, according to the bound task information, collecting, filtering and storing the flow data proxy, and intercepting the non-task target flow to obtain target flow proxy data.

In an implementation, a proxy server obtains a plurality of traffic request data. Then, the proxy server screens the plurality of flow request data based on the target task information to obtain initial flow request data. And the proxy server filters the initial flow request data according to a preset matching rule to obtain target flow request data.

Step 1605, receiving task information to be ended sent by the vulnerability discovery platform, and storing the task information to be ended into a pre-agent ending task queue.

In implementation, the distribution server receives task information to be ended sent by the vulnerability discovery platform, and stores the task information to be ended to a pre-agent ending task queue.

In step 1606, the target task information to be ended is obtained, the binding between the task and the proxy service is released, the task state is updated, the collected task traffic is packed, and the task traffic is issued to the task queue to be scanned.

In an implementation, the distribution server marks the target flow request data to obtain a label of the target flow data. Then, the distributing server reads the target task information to be ended in the pre-agent ending task queue, and releases the corresponding relation between the target task information and the address of the target agent node. Then, the distribution server updates the processing state of the target task information in the pre-proxy task queue based on the target to-be-ended information. And the distribution server packages the label corresponding to the target flow request data to obtain the downlink information. After determining the descending information, the distribution server stores the descending information into a preset task queue to be scanned.

In addition, the whole flow agent system is divided into a database storage module, a back-end task distribution module, mitmproxy flow agent nodes and a flow packing module. The function of each module is shown in table 13 below.

TABLE 13

Wherein the task distribution module (number 001) is used for scanning task distribution management. Mitmproxy traffic proxy node (number 002) is used for Mitmproxy traffic grabbing, filtering, and statistics operation encapsulation. The flow packing module (number 003) is used for periodically inquiring the task state, packing the task flow and pushing the scanning result. A database storage module (number 004) is used for scanning data storage.

In an alternative embodiment, the flow agent system is also provided with an error handling procedure, as shown in Table 14 below.

TABLE 14

If the task creation fails, the flow agent system returns a task creation failure message to the console so as to instruct the console to output. If the task stops failing, the flow agent system returns a task stop failure message to the console so as to instruct the console to output. If the task deletion fails, the flow agent system returns a task stop output message to the console so as to instruct the console to output.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a flow agent device for realizing the above-mentioned related flow agent method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the flow agent device provided below may be referred to the limitation of the flow agent method hereinabove, and will not be repeated here.

In one embodiment, as shown in fig. 17, there is provided a flow agent apparatus 1700 comprising: a ranking module 1701, a distribution module 1702, and a sending module 1703, wherein:

the ordering module 1701 is configured to receive the plurality of task information sent by the vulnerability discovery platform, and order the plurality of task information in the task processing queue based on the priority of the task information.

The distributing module 1702 is configured to obtain a state of each proxy node, and distribute, in a case where a target proxy node in an idle state exists, target task information in a task processing queue to the target proxy node; the target task information is used to instruct the target proxy node to determine target traffic request data.

And the sending module 1703 is configured to construct the sending information based on the determined target flow request data, and send the sending information to the vulnerability discovery platform.

In an exemplary embodiment, the ranking module 1701 includes a first receiving sub-module and a first ranking sub-module. Wherein the first ordering sub-module comprises:

the first storage sub-module is used for storing a plurality of task information into a preset initial task processing queue.

And the second sequencing sub-module is used for sequencing each task information in the initial task processing queue sequentially based on the sequence from high to low of the priority of the preset task information to obtain a task processing queue.

In an exemplary embodiment, the distribution module 1702 includes an acquisition sub-module and a distribution sub-module. Wherein the first distribution submodule includes:

and the first determining submodule is used for determining the first task information in the task processing queue as target task information under the condition that the idle target agent node exists.

And the first sending submodule is used for sending the target task information to the target proxy node.

And the establishing sub-module is used for establishing the corresponding relation between the address of the target agent node and the target task information.

In an exemplary embodiment, the transmitting module 1703 includes:

and the marking sub-module is used for marking the target flow request data to obtain the label of the target flow data.

And the packing sub-module is used for packing labels corresponding to the target flow request data to obtain the downlink information.

And the second sending submodule is used for storing the descending information into a preset task queue to be scanned and sending the descending information to the vulnerability mining platform according to the sequence of the task queue to be scanned.

In an exemplary embodiment, the flow agent apparatus 1700 includes, prior to execution of the packing submodule:

the second receiving sub-module is used for receiving the task information to be ended sent by the vulnerability mining platform and storing the task information to be ended into an ending task queue.

And the release sub-module is used for reading the target task information to be ended in the task ending queue, releasing the corresponding relation between the target task information and the address of the target agent node, and updating the processing state of the target task information in the task processing queue based on the target task information to be ended.

In an exemplary embodiment, there is provided a flow agent system comprising:

the distribution server is used for receiving the plurality of task information sent by the vulnerability discovery platform, and sequencing the plurality of task information in the task processing queue based on the priority of the task information; acquiring the state of each proxy node, and distributing target task information in a task processing queue to target proxy nodes under the condition that the idle state of the target proxy nodes exists; the target task information is used for indicating the target proxy node to determine target flow request data; constructing issuing information based on the determined target flow request data, and sending the issuing information to a vulnerability discovery platform;

In an exemplary embodiment, a flow proxy system is provided, where a proxy server is specifically configured to obtain a plurality of flow request data; screening the multiple flow request data based on the target task information to obtain initial flow request data; and filtering the initial flow request data according to a preset matching rule to obtain target flow request data.

The various modules in the above-described flow agent apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 18. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing flow agent data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program when executed by a processor implements a flow agent method.

It will be appreciated by those skilled in the art that the structure shown in fig. 18 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application is applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.

In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.

Claims

1. A method of flow agent, the method comprising:

2. The method of claim 1, wherein ordering a plurality of the task information in a task processing queue based on a priority of the task information, comprises:

3. The method of claim 1, wherein the distributing the target task information in the task processing queue to the target agent node in the presence of an idle state target agent node comprises:

transmitting the target task information to the target proxy node;

4. The method of claim 1, wherein the constructing the outbound message based on the determined target traffic request data and sending the outbound message to the vulnerability discovery platform comprises:

5. The method of claim 4, wherein said packaging the label corresponding to the target traffic request data, before obtaining the downlink message, further comprises:

6. A flow agent system, the system comprising:

7. The system of claim 6, wherein the proxy server is specifically configured to obtain a plurality of traffic request data;

8. A flow agent apparatus, the apparatus comprising:

9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.

10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.