CN116015844A

CN116015844A - Data flow detection method, system and electronic equipment

Info

Publication number: CN116015844A
Application number: CN202211668412.3A
Authority: CN
Inventors: 周林
Original assignee: Tianyi Safety Technology Co Ltd
Current assignee: Tianyi Safety Technology Co Ltd
Priority date: 2022-12-23
Filing date: 2022-12-23
Publication date: 2023-04-25

Abstract

The application provides a data traffic detection method, a data traffic detection system and electronic equipment, and relates to the technical field of network security. In the method, first service data meeting the characteristic conditions of first abnormal data are screened out from original flow data, the first service data are removed, then abnormal flow data meeting the characteristic conditions of second abnormal data are screened out from first residual flow data, and initial detection of the original flow is achieved, so that follow-up accurate detection of the flow of the original flow data can be reduced, calculation amount of data with abnormal flow is screened out, resource expenditure of a system is reduced, and efficiency of detecting the data flow can be improved.

Description

Data flow detection method, system and electronic equipment

Technical Field

The present invention relates to the field of network security technologies, and in particular, to a method and a system for detecting data traffic, and an electronic device.

Background

With the development of internet technology, the network bandwidth is increased, and the network with high-speed wide connection provides convenience for the life of people and also provides favorable conditions for network attack.

For a common distributed denial of service attack (Distributed Denial of Service, DDoS), an attacker typically uses a network protocol vulnerability or a system vulnerability to control a proxy host to send a large number of false attack packets to a server, which results in blocking the network bandwidth of the server, and flooding a large number of false attack packets with normal service request data packets, which results in the server failing to provide normal services for users.

At present, in order to defend network attack, ensure that a server can provide stable and normal service, a cloud anti-clustering technology can be adopted to detect data traffic and screen abnormal traffic data. Firstly, the cleaning equipment pulls flow data from a message service end, then carries out data cleaning on the flow data, then sends the flow data obtained after the data cleaning to a flow type calculation queue for flow type calculation, judges whether the flow of the data is normal in real time according to the calculation result, and finally sends the data with normal flow to a server to provide normal service for a user.

By adopting the mode, the process of pulling the flow data from the message service end by the cleaning equipment can increase the link length of the defense system, and the flow data is totally sent to the flow calculation queue for flow calculation, so that the resource cost of the defense system can be increased, and the efficiency of detecting the flow of the data can be reduced.

Disclosure of Invention

The invention provides a data flow detection method, a system and electronic equipment, which are used for reducing the resource overhead of the system and improving the efficiency of detecting the data flow. The specific technical scheme is as follows:

in a first aspect, the present application provides a method for detecting data traffic, including:

receiving original flow data, and carrying out feature extraction on the original flow data to obtain a corresponding data feature set; wherein the original flow data at least comprises one kind of service data;

determining that the flow of the original flow data is abnormal when a first data feature meeting a first abnormal data feature condition exists in the data feature set;

removing first service data corresponding to the first data characteristic from the original flow data to obtain first residual flow data;

determining that the flow of the first residual flow data is abnormal when a second data feature meeting a second abnormal data feature condition exists in the data feature set of the first residual flow data; wherein the first data feature extraction complexity is greater than the second data feature.

Based on the method, the original flow data can be subjected to hierarchical data cleaning, and preliminary screening of the original flow data is realized.

In one possible design, after the determining of the flow anomaly of the first residual flow data, the method further includes:

removing second service data corresponding to the second data features from the first residual flow data to obtain second residual flow data;

when determining that a third data feature meeting a chained computing mode exists in the data feature set of the second residual flow data, performing chained computing on the second residual flow data based on the chained computing mode, and determining third business data with abnormal flow in the second residual flow data, wherein the third data feature extraction complexity is between the first data feature and the second data feature;

and removing the third service data from the second residual flow data to obtain target service data, and taking the target service data as normal flow data.

Based on the method, accurate flow calculation can be performed on the second residual flow data, abnormal flow in the second residual flow data can be screened out, and normal flow data can be obtained.

In one possible design, if there is no third data feature satisfying the chained mode of calculation in the data feature set of the second residual flow data;

judging whether the second residual flow data carries a flow calculation identifier or not;

if the second residual flow data carries a flow calculation identifier, the second residual flow data is imported into a flow cluster queue for flow calculation, and third business data with abnormal flow in the second residual flow data is determined;

Based on the method, the second residual flow data can be selectively imported into the flow cluster queue for flow calculation, and the normal flow data is finally obtained.

In one possible design, if the second residual traffic data does not carry a flow calculation identifier, the third service data is removed from the second residual traffic data to obtain target service data, and the target service data is used as normal traffic data.

Based on the method, the third service data can be directly removed from the second residual flow data to obtain the target service data, and the target service data is used as the normal flow data.

In a second aspect, the present application provides a data traffic detection system comprising:

the receiving module is used for receiving the original flow data, and extracting the characteristics of the original flow data to obtain a corresponding data characteristic set; wherein the original flow data at least comprises one kind of service data;

the detection module is used for determining flow abnormality of the original flow data when the first data feature meeting the first abnormal data feature condition exists in the data feature set;

In one possible design, the detection module is further configured to:

and if the second residual flow data does not carry the flow calculation identification, removing the third service data from the second residual flow data to obtain target service data, and taking the target service data as normal flow data.

In a third aspect, the present application provides an electronic device, including:

a memory for storing a computer program;

and the processor is used for realizing the steps of the data flow detection method when executing the computer program stored in the memory.

In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored therein, which when executed by a processor, implements the steps of the data traffic detection method described above.

The technical effects of each of the second to fourth aspects and the technical effects that may be achieved by each aspect are referred to above for the technical effects that may be achieved by the first aspect or each possible aspect in the first aspect, and the detailed description is not repeated here.

Drawings

Fig. 1 is a flowchart of a data traffic detection method provided in the present application;

FIG. 2 is a schematic diagram of a data traffic detection system architecture provided herein;

FIG. 3 is a schematic diagram of a data flow detection system according to the present disclosure;

fig. 4 is a schematic structural diagram of an electronic device provided in the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings. The specific method of operation in the method embodiment may also be applied to the device embodiment or the system embodiment. It should be noted that "a plurality of" is understood as "at least two" in the description of the present application. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. A is connected with B, and can be represented as follows: both cases of direct connection of A and B and connection of A and B through C. In addition, in the description of the present application, the words "first," "second," and the like are used merely for distinguishing between the descriptions and not be construed as indicating or implying a relative importance or order.

Embodiments of the present application are described in detail below with reference to the accompanying drawings.

In view of this, in order to reduce resource overhead of the defense system and improve efficiency of detecting data traffic, the present application provides a data traffic detection method, which specifically includes: firstly, receiving original flow data, carrying out feature extraction on the original flow data to obtain a corresponding data feature set, then determining flow abnormality of the original flow data when a first data feature meeting a first abnormal data feature condition exists in the data feature set, then removing first service data corresponding to the first data feature from the original flow data to obtain first residual flow data, and finally determining flow abnormality of the first residual flow when a second data feature meeting a second abnormal data feature condition exists in the data feature set of the first residual flow data.

It is easy to see that, through the method, the first service data meeting the characteristic condition of the first abnormal data can be screened out from the original flow data, the first service data is removed, then the abnormal flow data meeting the characteristic condition of the second abnormal data is screened out from the first residual flow data, and the primary detection of the original flow is realized, so that the subsequent accurate detection of the flow of the original flow data can be reduced, the calculated amount of the flow data with abnormal flow is screened out, the resource cost of a system is reduced, and the detection efficiency of the data flow can be improved.

Referring to fig. 1, a flow chart of a data flow detection method provided in an embodiment of the present application is shown, where the method includes:

s1, receiving original flow data, and carrying out feature extraction on the original flow data to obtain a corresponding data feature set;

in the first place, the method provided by the application can be applied to the system architecture shown in fig. 2, and the system architecture comprises a service request server 1, a service request server 2, a message server, a cleaning device, a target server and a streaming computing cluster.

Illustratively, in the embodiments of the present application, the cleaning device may be deployed in a message server, and the streaming computing cluster may be a separate computing unit.

The number of the devices is not limited in this embodiment, and as shown in fig. 2, only the service request server 1, the service request server 2, the message server, the cleaning device, the target server, and the streaming computing cluster are described as examples. The above-described respective devices and their respective functions are briefly described below.

The service request server is used for sending a corresponding service request to the target server; the message server is used for detecting the flow of the original flow data contained in the service request, and after receiving the original flow data, the message server sends the original flow data to the data cleaning equipment, and designs different data cleaning modes according to the complexity of the feature extraction of the original flow data so as to realize the primary screening of the original flow data; the message server obtains first residual flow data after removing abnormal original flow data in the original flow data, in order to further accurately detect the first residual flow data, firstly, extracting complexity according to the characteristics of the first residual flow data, setting different calculation modes, then, carrying out chain calculation on flow data meeting the condition of the chain calculation mode in the first residual flow data, determining flow abnormal flow data in the first residual flow data according to the chain calculation result, removing abnormal flow data in the first residual flow data to obtain second residual flow data, finally, selectively guiding the second flow data into a streaming calculation cluster to carry out calculation, determining abnormal flow data in the second residual flow data according to the streaming calculation result, realizing accurate detection of the flow of the original flow data, and finally, sending the obtained normal flow data to a target processor; the target server is used for receiving the normal flow data and providing the service corresponding to the service request for the user; the stream computing cluster is used for computing the data traffic and determining abnormal traffic.

In this embodiment of the present application, first, the message server receives original traffic data, where the original traffic data may be from a traffic detection device or some target terminals (users) that have established corresponding connections with the target server, and the original traffic data includes multiple kinds of service data, may be some normal service data, and some abnormal service data, such as attack traffic, and the kind of service data is not specifically limited in this application and will not be described herein.

And then the message server can design different feature extraction modes according to different service data in the original flow data, and perform feature extraction on the original flow data to obtain a corresponding data feature set, for example, feature information, source address, target address, abnormal access identifier, access times and flow size contained in a flow log of the original flow data.

By designing different feature extraction modes for different service data in the original flow data, a data feature set corresponding to the original flow data can be obtained.

S2, determining that the flow of the original flow data is abnormal when the first data feature meeting the first abnormal data feature condition exists in the data feature set.

In this embodiment of the present application, after obtaining a data feature set corresponding to original traffic data, the message server may first determine whether there is a first data feature that satisfies a first abnormal data feature condition in the data feature set, where, by way of example, the first data feature may be a source IP of the original traffic data or some abnormal access identifiers, and when it is determined that a source address traced back by the source IP does not satisfy a secure access condition or the original traffic data has been marked as an abnormal access traffic, then it may be determined that the traffic of the original traffic data is abnormal.

For example, some simple logic rules may be set for the original flow data, to determine whether the variable value of the original flow data is within a reasonable value range, for example, when the weight is tested, and when the weight is negative, the abnormal flow of the original flow data may be determined.

By the method, the original flow data can be subjected to preliminary data cleaning by judging whether the source IP of the original flow data has abnormal access marks or not.

S3, removing the first service data corresponding to the first data characteristic from the original flow data to obtain first residual service data.

In the embodiment of the application, when the message server determines that the source IP of the original traffic data does not meet the security access condition or that the original traffic data has an abnormal access identifier, the message server may screen out first service data corresponding to the first data feature to obtain first residual traffic data. The first service data may be a plurality of service data including similar first data features, for example, service data 1 and service data 2 with abnormal access identifiers, and the type of the service data is not specifically limited in this application, and will not be described herein.

By the method, the first service data corresponding to the first data characteristics can be screened out, and the first residual flow data can be obtained.

And S4, determining that the flow of the first residual flow data is abnormal when the second data features meeting the second abnormal data feature conditions exist in the data feature set of the first residual flow data.

After the message server obtains the first residual flow data, firstly judging whether a second data feature meeting a second abnormal data feature condition exists in a data feature set of the first residual flow data, wherein the second data feature can be a data feature which is mutually related in the data feature set of the first residual flow data, and the abnormal flow data cannot be judged by combining the data feature set of a service library or through a source IP corresponding to the first residual flow data.

In determining that there is a second data feature in the set of data features of the first remaining flow data that satisfies the second abnormal data feature condition, a flow anomaly of the first remaining flow data may be determined, and illustratively, the abnormal flow data in the first remaining flow data may be determined in combination with big data, or some specific algorithm.

By the method, the first residual flow data can be further subjected to data cleaning, the primary screening of the original flow data is realized in the message server, and the link length of data transmission is reduced.

In a possible implementation manner, after performing step 4, removing second service data corresponding to the second data feature from the first residual traffic data to obtain second residual traffic data;

when the third data characteristic meeting the chained calculation mode exists in the data characteristic set of the second residual flow data, chained calculation is carried out on the second residual flow data based on the chained calculation mode, and third service data with abnormal flow in the second residual flow data is determined;

In this embodiment of the present application, after removing the second service data corresponding to the second data feature from the first residual traffic data, the message server may obtain second residual traffic data, and in order to further perform accurate traffic detection on the second residual traffic data, it needs to dynamically combine the data feature of the second residual traffic data in real time, and perform streaming computation on the second residual traffic data.

Before the message server performs streaming computation on the second residual flow data, it may first determine whether a third data feature satisfying the chained computation mode exists in the data feature set of the second residual flow data, where the third data feature may be that there is no data feature associated with the data feature set of the second residual flow data, that is: the service data contained in the second residual flow data are mutually independent.

For example, if there is a third data feature satisfying the chained computation manner in the data feature set of the second residual traffic data, the message server may perform chained computation on the second residual traffic data according to the chained computation manner, for example, whether the source IP of the second residual traffic data meets the condition of secure access, carries an abnormal access identifier, whether the traffic size is at a set security threshold, whether the resource utilization rate of each process is in a reasonable configuration interval, and other integrated processing logic to perform chained computation on the traffic of the second residual traffic data, then determine, according to the chained computation result, third traffic data with abnormal traffic in the second residual traffic data, and finally remove the third traffic data from the second residual traffic data to obtain target traffic data, and may send the target traffic data as normal traffic data to the target server. Here, it should be noted that the third service data may be data having a high requirement for timeliness, for example, the number of times of occurrence of attack traffic in a set time range; the chained computation can be deployed in a message server for direct processing, and the integrated processing logic is similar to a pipeline working mode and can be added in a self-defined manner according to the level of service safety, and is not described in detail herein.

If the third data feature satisfying the chained computing manner does not exist in the data feature set of the second residual traffic data, the message server may first determine whether the second residual traffic data carries a streaming computing identifier, where the streaming computing identifier is set for a requirement level of different users, and is not described in detail herein, and then if the second residual traffic data carries the streaming computing identifier, the second residual traffic data is imported into a traffic cluster queue for streaming computing, where the streaming computing queue may be a Flink cluster, and the Flink cluster may also perform relatively accurate streaming computing for the data feature correlated with the data feature existing in the data feature set of the second residual traffic data.

The message server imports the second residual flow data into the Flink cluster to obtain a corresponding calculation result, the third service data with abnormal flow in the second residual flow data can be accurately determined according to the calculation result, the third service data is removed from the second residual flow data to obtain target service data, and finally the target service data is sent to the target server as normal flow data to provide normal service for a user.

And if the second residual flow data does not carry the flow type calculation identifier, removing the third service data from the second residual flow data to obtain target service data, and finally sending the target service data to the target server as normal flow data to provide normal service for the user.

In summary, according to the data traffic detection method provided by the embodiment of the application, data cleaning can be performed on traffic data with simpler data feature extraction in original traffic data in a message server, primary screening of the original traffic data is achieved, then streaming calculation is performed on residual traffic data obtained after primary screening, the residual traffic data is divided into a chained calculation mode and a streaming calculation mode according to different feature extraction difficulties, the chained calculation mode can be directly calculated in the message server, the streaming calculation mode needs to be processed in a cluster queue deployed independently, finally traffic data with abnormal traffic in the residual traffic data are screened out according to a calculation result, target traffic data are obtained, the target traffic data are used as normal traffic data, and the normal service is provided for users in a target processor.

Based on the method provided in the foregoing embodiment, the embodiment of the present application further provides a data traffic detection system, as shown in fig. 3, which is a schematic structural diagram of a data traffic detection system in the embodiment of the present application, where the system includes:

the receiving module 301 is configured to receive original flow data, and perform feature extraction on the original flow data to obtain a corresponding data feature set; wherein the original flow data at least comprises one kind of service data;

a detection module 302, configured to determine, when it is determined that a first data feature that meets a first abnormal data feature condition exists in the data feature set, that a flow of the original flow data is abnormal;

In one possible design, the detection module 302 is further configured to:

Based on the same inventive concept, the embodiment of the present application further provides an electronic device, where the electronic device may implement the function of the foregoing data traffic detection method, and referring to fig. 4, the electronic device includes:

at least one processor 401, and a memory 402 connected to the at least one processor 401, in this embodiment of the present application, a specific connection medium between the processor 401 and the memory 402 is not limited, and in fig. 4, the processor 401 and the memory 402 are connected by a bus 400 as an example. The bus 400 is shown in bold lines in fig. 4, and the manner in which the other components are connected is illustrated schematically and not by way of limitation. The bus 400 may be divided into an address bus, a data bus, a control bus, etc., and is represented by only one thick line in fig. 4 for ease of illustration, but does not represent only one bus or one type of bus. Alternatively, the processor 401 may be referred to as a controller, and the name is not limited.

In the embodiment of the present application, the memory 402 stores instructions executable by the at least one processor 401, and the at least one processor 401 may execute the data traffic detection method described above by executing the instructions stored in the memory 402. Processor 401 may implement the functions of the various modules in the system shown in fig. 3.

The processor 401 is a control center of the apparatus, and various interfaces and lines can be used to connect various parts of the entire control device, and by executing or executing instructions stored in the memory 402 and invoking data stored in the memory 402, various functions of the apparatus and processing data can be performed, so that the apparatus is monitored as a whole.

In one possible design, processor 401 may include one or more processing units, and processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes operating systems, user interfaces, application programs, and the like, and the modem processor primarily processes wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 401. In some embodiments, processor 401 and memory 402 may be implemented on the same chip, and in some embodiments they may be implemented separately on separate chips.

The processor 401 may be a general purpose processor such as a Central Processing Unit (CPU), digital signal processor, application specific integrated circuit, field programmable gate array or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, which may implement or perform the methods, steps and logic blocks disclosed in the embodiments of the present application. The general purpose processor may be a microprocessor or any conventional processor or the like. The steps of the data flow detection method disclosed in connection with the embodiments of the present application may be directly embodied in a hardware processor for execution, or may be executed by a combination of hardware and software modules in the processor.

Memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The Memory 402 may include at least one type of storage medium, which may include, for example, flash Memory, hard disk, multimedia card, card Memory, random access Memory (Random Access Memory, RAM), static random access Memory (Static Random Access Memory, SRAM), programmable Read-Only Memory (Programmable Read Only Memory, PROM), read-Only Memory (ROM), charged erasable programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory), magnetic Memory, magnetic disk, optical disk, and the like. Memory 402 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 402 in the present embodiment may also be circuitry or any other device capable of implementing a memory function for storing program instructions and/or data.

By programming the processor 401, the code corresponding to the data flow rate detection method described in the foregoing embodiment may be cured into the chip, so that the chip can execute the steps of the data flow rate detection method of the embodiment shown in fig. 1 at the time of operation. How to design and program the processor 401 is a technology well known to those skilled in the art, and will not be described in detail here.

Based on the same inventive concept, the embodiments of the present application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the data traffic detection method discussed above.

In some possible embodiments, aspects of the data traffic detection method provided herein may also be implemented in the form of a program product comprising program code for causing a control apparatus to carry out the steps of the data traffic detection method according to various exemplary embodiments of the present application as described herein above when the program product is run on a device.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present application without departing from the spirit or scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims and the equivalents thereof, the present application is intended to cover such modifications and variations.

Claims

1. A method for detecting data traffic, the method comprising:

2. The method of claim 1, wherein after the determining of the traffic anomaly of the first remaining traffic data, further comprising:

3. The method of claim 2, wherein if there is no third data feature satisfying the chained mode of computation in the set of data features of the second residual flow data;

4. The method of claim 3 wherein if the second remaining traffic data does not carry a flow calculation identifier, removing the third traffic data from the second remaining traffic data to obtain target traffic data, and using the target traffic data as normal traffic data.

5. A data traffic detection system, comprising:

6. The system of claim 5, wherein the detection module is further to:

7. The system of claim 5, wherein the detection module is further to:

8. The system of claim 5, wherein the detection module is further to:

9. An electronic device, comprising:

a memory for storing a computer program;

a processor for carrying out the method steps of any one of claims 1-4 when executing a computer program stored on said memory.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored therein a computer program which, when executed by a processor, implements the method steps of any of claims 1-4.