CN110138720B - Method and device for detecting abnormal classification of network traffic, storage medium and processor - Google Patents
Method and device for detecting abnormal classification of network traffic, storage medium and processor Download PDFInfo
- Publication number
- CN110138720B CN110138720B CN201910217643.4A CN201910217643A CN110138720B CN 110138720 B CN110138720 B CN 110138720B CN 201910217643 A CN201910217643 A CN 201910217643A CN 110138720 B CN110138720 B CN 110138720B
- Authority
- CN
- China
- Prior art keywords
- target
- attribute value
- value
- target attribute
- field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
Abstract
The invention discloses a method and a device for detecting abnormal classification of network flow, a storage medium and a processor. The method comprises the following steps: acquiring a target log generated by a target website under target network traffic; acquiring an attribute value set of a target log, wherein an attribute value in the attribute value set is used for indicating the state of a field associated with the attribute value under a target attribute; synthesizing at least one target attribute value associated with the same field into a first target attribute value if the set of attribute values includes a predefined target attribute value; and under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, determining that the target network traffic has an abnormal state of a target type, wherein the second target attribute value is used for indicating that a field associated with the second target attribute value belongs to the abnormal state of the target type. By the method and the device, the technical effect of improving the efficiency of classifying and detecting the abnormal traffic of the website is achieved.
Description
Technical Field
The invention relates to the field of internet, in particular to a method and a device for detecting abnormal classification of network flow, a storage medium and a processor.
Background
Currently, in network traffic, there is usually abnormal traffic, for example, there is about 25% abnormal traffic. Among these traffic, there are different classifications, some for brush volume, some for crawlers, and some for frequent visits, and these abnormal traffic belong to invalid traffic visited by the user. Therefore, a classification detection of the abnormal traffic of the website is required.
In the related art, the type of abnormal website traffic is usually observed manually, but the timeliness of analyzing the website traffic is poor, and manpower is consumed; in the related technology, the type of the newly added network traffic is judged through the historical traffic sequence of each site, so that the accumulation of the historical traffic is needed, the storage space is increased, the network abnormal traffic cannot be classified and detected in time, and the technical problem of low efficiency in classifying and detecting the abnormal traffic of the website exists.
Aiming at the technical problem of low efficiency of classifying and detecting the abnormal traffic of the network station in the prior art, no effective solution is provided at present.
Disclosure of Invention
The invention mainly aims to provide a method, a device, a storage medium and a processor for detecting network traffic abnormality by classification, so as to at least solve the technical problem of low efficiency of detecting website abnormal traffic by classification.
In order to achieve the above object, according to one aspect of the present invention, a method for detecting an abnormal classification of network traffic is provided. The method comprises the following steps: acquiring a target log generated by a target website under target network traffic; acquiring an attribute value set of a target log, wherein an attribute value in the attribute value set is used for indicating the state of a field associated with the attribute value under a target attribute; synthesizing at least one target attribute value associated with the same field into a first target attribute value if the set of attribute values includes a predefined target attribute value; and under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, determining that the target network traffic has an abnormal state of a target type, wherein the second target attribute value is used for indicating that a field associated with the second target attribute value belongs to the abnormal state of the target type.
Optionally, synthesizing at least one target attribute value associated with the same field into the first target attribute value comprises: under the condition that a plurality of target attribute values are associated with the same field, carrying out logic OR processing on binary values of the plurality of target attribute values associated with the same field to obtain a first target attribute value; in the case where there is one target attribute value associated with the same field, the target attribute value is determined as the first target attribute value.
Optionally, before determining that the target abnormal state exists in the target network traffic, the method further includes: traversing a plurality of second target attribute values associated with the exception state of the target type; performing logic and processing on the binary value of the first target attribute value and the binary value of a second target attribute value traversed currently to obtain a target processing result; when the target processing result is larger than the target value, determining that the first target attribute value is successfully matched with the traversed second target attribute value; and determining the next second target attribute value in the plurality of second target attribute values as the currently traversed second target attribute value under the condition that the target processing result is not larger than the target value and the plurality of second target attribute values are not traversed.
Optionally, after synthesizing at least one target attribute value associated with the same field into the first target attribute value, the method further comprises: storing the first target attribute value in a target log.
Optionally, obtaining the set of attribute values of the target log comprises: acquiring a plurality of target fields in a target log; a set including attribute values respectively associated with the plurality of target fields is determined as a set of attribute values.
In order to achieve the above object, according to another aspect of the present invention, an apparatus for detecting an abnormal classification of network traffic is also provided. The device includes: the first acquisition unit is used for acquiring a target log generated by a target website under the target network flow; the second acquisition unit is used for acquiring the attribute value set of the target log, wherein the attribute value in the attribute value set is used for indicating the state of the field associated with the attribute value under the target attribute; a synthesizing unit for synthesizing at least one target attribute value associated with the same field into a first target attribute value if the set of attribute values includes a predefined target attribute value; the first determining unit is used for determining that the target network traffic has an abnormal state of a target type under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, wherein the second target attribute value is used for indicating that a field associated with the second target attribute value belongs to the abnormal state of the target type.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a storage medium. The storage medium comprises a stored program, wherein the apparatus on which the storage medium is located is controlled to perform the method of the embodiments of the present invention when the program is run.
To achieve the above object, according to another aspect of the present invention, there is also provided a processor. The processor is configured to run a program, wherein the program performs the method of the embodiment of the present invention when running.
According to the invention, a target log generated by a target website under the target network flow is acquired; acquiring an attribute value set of a target log, wherein an attribute value in the attribute value set is used for indicating the state of a field associated with the attribute value under a target attribute; synthesizing at least one target attribute value associated with the same field into a first target attribute value if the set of attribute values includes a predefined target attribute value; and under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, determining that the target network traffic has an abnormal state of a target type, wherein the second target attribute value is used for indicating that a field associated with the second target attribute value belongs to the abnormal state of the target type. That is to say, the target attribute values are predefined, under the condition that the attribute value set of the target log includes the predefined target attribute value, at least one target attribute value is synthesized into a first target attribute value (synthesized attribute value), the first target attribute value is matched with a predefined second target attribute value which is classified well, and under the condition that the matching is successful, the target network traffic is determined to have an abnormal state of the target type, so that the technical problem that the efficiency of classifying and detecting the abnormal traffic of the website is low is solved, and the technical effect of improving the efficiency of classifying and detecting the abnormal traffic of the website is achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
fig. 1 is a flowchart of an abnormal classification detection method of network traffic according to an embodiment of the present invention. (ii) a
FIG. 2 is a flow diagram of a method of exception-classified storage according to an embodiment of the present invention;
FIG. 3 is a flow chart of a decomposition matching method for anomaly classification according to an embodiment of the present invention; and
fig. 4 is a schematic diagram of an apparatus for detecting abnormal classification of network traffic according to an embodiment of the present invention.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
The embodiment of the invention provides an abnormal classification detection method of network flow.
Fig. 1 is a flowchart of an abnormal classification detection method of network traffic according to an embodiment of the present invention. As shown in fig. 1, the method comprises the steps of:
step S102, a target log generated by the target website under the target network flow is obtained.
In the technical solution provided in step S102, the target network traffic is traffic to be detected whether abnormal or not. The target log, that is, the website log, is generated for the target website under the target network traffic. The target log is a file ending in a log which records various kinds of original information such as processing requests received by the server and server operation errors, can record information such as operation conditions and access requests of a target website in the operation process, and can clearly determine information such as what IP, what time, what operating system, what browser, what resolution display and the like a user accesses to which page of the target website and whether the access is successful or not.
Alternatively, the target log of this embodiment may be represented by a log ID, for example, the target log is represented by xxx _001, xxx _002, xxx _003, or the like.
Step S104, obtaining the attribute value set of the target log.
In the technical solution provided in step S104, after a target log generated by the target website under the target network traffic is obtained, an attribute value set of the target log is obtained, where an attribute value in the attribute value set is used to indicate a state of a field associated with the attribute value under the target attribute.
In this embodiment, the target log includes fields, such as a field a corresponding to a cookie, a field B corresponding to an Internet Protocol (IP), a field C corresponding to a User Agent (User Agent), etc., wherein the User Agent may be a browser, and may further include fields such as a timestamp, a web arrival (referrer), etc., each of which may have a plurality of attribute values corresponding to different data presentations. The attribute value set of the target log comprises an attribute value obtained by calculating a field of the target log, wherein the attribute value is used for indicating the state of the target log under the field associated with the attribute value, and the state can be a normal state or an abnormal state. Optionally, the attribute value of the a field is 1, which is used to indicate that the state of the a field under the cookie attribute is a cookie format abnormal state, the attribute value of the a field is 2, which is used to indicate that the state of the a field under the cookie attribute is an exposure abnormal state, the attribute value of the B field is 16, which is used to indicate that the state of the B field under the IP attribute is an IP change too fast state, and the like, which is not limited herein.
Step S106, synthesizing at least one target attribute value associated with the same field into a first target attribute value, in case the set of attribute values comprises a predefined target attribute value.
In the technical solution provided in step S106, after obtaining a target log generated by a target website under a target network traffic, if the attribute value set includes a predefined target attribute value, synthesizing at least one target attribute value associated with the same field into a first target attribute value.
In this embodiment, the attribute value of the exception field is predefined to obtain a target attribute value. For example, for field a corresponding to a cookie, the rule defines the value a: 1 is used for indicating that the state of the field A under the cookie attribute is cookie format exception under the condition that the target attribute value is 1; the rule defines the value a: 2, used for indicating that the state of the field A under the cookie attribute is cookie exposure abnormity under the condition that the target attribute value is 2; the rule defines the value a: 4, used for showing that the state of the field A under the cookie attribute is cookie click abnormity under the condition that the target attribute value is 4; the rule defines the value a:8, used for indicating that the state of the field A under the cookie attribute is that the cookie changes too fast under the condition that the target attribute value is 8; the rule defines the value a: and 16, for indicating that the state of the field a under the cookie attribute is cookie time-out in case that the target attribute value is 16. Wherein the rule definition value is a condition defined by the anomaly classification.
Optionally, for field B corresponding to IP, the rule defines the value B:1, which is used for indicating that the state of the field B under the IP attribute is crawler IP abnormity under the condition that the target attribute value is 1; the rule defines the value B: 2, the field B is used for indicating that the state of the field B under the IP attribute is the data center IP anomaly if the target attribute value is 2; the rule defines the value B: 4, used for showing that the state of the field B under the IP attribute is proxy IP abnormal under the condition that the target attribute value is 4; the rule defines the value B: 8, used for showing that the state of the field B under the IP attribute is a standby IP exception under the condition that the target attribute value is 8; the rule defines the value B: and 16, for indicating that the state of the field B under the IP attribute is that the IP changes too fast when the target attribute value is 16.
Optionally, for field C corresponding to the usergent, the rule defines the value C:1 is used for indicating that the state of the field C under the Useragent attribute is simple crawler Useragent under the condition that the target attribute value is 1; the rule defines the value C: 2 is used for indicating that the state of the field C under the Useragent attribute is Useragent too short under the condition that the target attribute value is 2; the rule defines the value C: 4 is used to indicate that the state of field C under the Useragent attribute is a high-level crawler Useragent, if the target attribute value is 4.
It should be noted that the predefined target attribute value is only an example of the embodiment of the present invention, and the predefined target attribute value that does not represent the embodiment of the present invention is only the above, and any predefined target attribute value for determining whether the website traffic is abnormal is within the scope of the embodiment, and is not illustrated here.
The embodiment determines the attribute values in the attribute value set, may perform matching calculation with the predefined target attribute value, and synthesizes at least one target attribute value associated with the same field into a first target attribute value in the case that the attribute value set includes the predefined target attribute value, where the first target attribute value is also the synthesized attribute value of the target log. For example, the attribute value set includes attribute values 1, 8, 16 of the a field, that is, the rule definition value a: 1, rule definition value a:8, rule definition value A: 16, synthesizing the target attribute values 1, 8 and 16 of the A field into a first target attribute value 25 of the A field; the attribute value set also includes an attribute value 1 of the C field, i.e., the corresponding rule definition value C:1, the first target attribute value may be the target attribute value 1 of the C field.
And step S108, determining that the target network flow has an abnormal state of the target type under the condition that the first target attribute value is successfully matched with the predefined second target attribute value.
In the technical solution provided in step S108, when the first target attribute value is successfully matched with the predefined second target attribute value, it is determined that the target network traffic has an abnormal state of the target type, where the second target attribute value is used to indicate that a field associated with the second target attribute value belongs to the abnormal state of the target type.
In this embodiment, a second target attribute value is predefined, which corresponds to the abnormal state of the target type to which the field belongs, i.e. the abnormal classification result. For example, if the abnormal state of the target type is a frequency abnormality, the second target attribute value may be an attribute value 8 in the field a, that is, the rule definition value a is: 8, the second target attribute value may also be an attribute value 16 in the B field, and the rule definition value B is: 16; for another example, if the abnormal state of the target type is a crawler problem, the rule definition value may be an attribute value 1 in a field B, corresponding to the rule definition value B:1, it may also be an attribute value 1 in the C field, and the corresponding rule defines a value C:1, it may also be an attribute value 4 under the C field, and the corresponding rule defines a value C: 4, for representing a set of various crawler questions.
It should be noted that the predefined second target attribute value and the abnormal state of the field associated with the second target attribute value belonging to the target type are merely an example of the embodiment of the present invention, and do not represent that the second target attribute value and the abnormal state of the field associated with the second target attribute value belonging to the target type of the embodiment of the present invention are merely the above examples, and any second target attribute value that can be predefined for determining whether the website traffic is abnormal and the abnormal state of the field associated with the second target attribute value belonging to the target type are within the scope of the embodiment, and are not illustrated here.
Under the condition that the first target attribute value is successfully matched with the predefined second target attribute value, for example, the binary value of the first target attribute value and the binary value of the predefined second target attribute value are logically and-processed, if the output result is greater than 0, it is determined that the target log has an abnormal state of the target type, and further it is determined that the target network traffic corresponding to the target log has an abnormal state of the target type, that is, an abnormal classification result is obtained, the purpose of classifying the abnormality of the target network traffic is achieved, and the abnormal classification result can be used in a report for finally counting the target network traffic.
For example, the first target attribute value of target log xxx _001 is 25 under the A field, i.e., A: 25, C:1, the matching with the predefined second target attribute value 1 in the a field, the predefined second target attribute value 8 in the a field, and the predefined second target attribute value 16 in the a field is successful, that is, the matching with the predefined second target attribute value 1 in the a field, that is, the predefined matching with the predefined second target attribute value 16 in the a field is successful, that is, the predefined matching with the predefined second target attribute value 1 in the a field is successful: 1. a: 8. a: 16, the exception status of the target type may be a: 1, and A:8, and a cookie time timeout exception classification result corresponding to a 16.
In this embodiment, the abnormal state of the target type may match a plurality of condition defined by the classification (rule definition value), for example, the frequent abnormal classification result may match a:8(cookie is changed too fast) and B:16(IP is changed too fast), and as long as the first target attribute value satisfies one of the condition defined by the classification, the abnormal state of the target type may be determined to be the abnormal classification corresponding to the condition defined by the classification.
In this embodiment, the target log simultaneously satisfies the conditions defined by different abnormal classifications, and then participates in different abnormal classification calculations in the final abnormal classification result.
In the embodiment, a target log generated by a target website under target network traffic is acquired; acquiring an attribute value set of a target log, wherein an attribute value in the attribute value set is used for indicating the state of a field associated with the attribute value under a target attribute; synthesizing at least one target attribute value associated with the same field into a first target attribute value if the set of attribute values includes a predefined target attribute value; and under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, determining that the target network traffic has an abnormal state of a target type, wherein the second target attribute value is used for indicating that a field associated with the second target attribute value belongs to the abnormal state of the target type. That is to say, the target attribute values are predefined, under the condition that the attribute value set of the target log includes the predefined target attribute value, at least one target attribute value is synthesized into a first target attribute value (synthesized attribute value), the first target attribute value is matched with a predefined second target attribute value which is classified well, and under the condition that the matching is successful, the target network traffic is determined to have an abnormal state of the target type, so that the technical problem that the efficiency of classifying and detecting the abnormal traffic of the website is low is solved, and the technical effect of improving the efficiency of classifying and detecting the abnormal traffic of the website is achieved.
As an alternative implementation, the step S106, synthesizing at least one target attribute value associated with the same field into a first target attribute value includes: under the condition that a plurality of target attribute values are associated with the same field, carrying out logic OR processing on binary values of the plurality of target attribute values associated with the same field to obtain a first target attribute value; in the case where there is one target attribute value associated with the same field, the target attribute value is determined as the first target attribute value.
In this embodiment, when a plurality of target attribute values associated with the same field are provided, a binary value of the plurality of target attribute values associated with the same field is logically or-processed to obtain a first target attribute value, that is, a binary value of a different target attribute value associated with the same field is logically or-processed to generate a new attribute value, where the first target attribute value is a composite attribute value in a target log, for example, if the target log includes an a field and a C field, the target attribute value associated with the a field is 1, 8, and 16, and corresponds to a rule definition value a: 1. a: 8. a: 16, the binary values of 1, 8, 16 associated with the a field are logically or-ed to obtain 25, which may be represented by a 1: and 25, respectively.
In a case where there is one target attribute value associated with the same field, the target attribute value is determined as the first target attribute value, and the target attribute value may be directly determined as the first target attribute value, for example, if the target attribute value associated with the C field is 1, then 1 is directly determined as the first target attribute value, and the ratio of C:1 is shown.
As an optional implementation manner, before determining that the target abnormal state exists in the target network traffic at step S108, the method further includes: traversing a plurality of second target attribute values associated with the exception state of the target type; performing logic and processing on the binary value of the first target attribute value and the binary value of a second target attribute value traversed currently to obtain a target processing result; when the target processing result is larger than the target value, determining that the first target attribute value is successfully matched with the traversed second target attribute value; and determining the next second target attribute value in the plurality of second target attribute values as the currently traversed second target attribute value under the condition that the target processing result is equal to 0 and the plurality of second target attribute values are not traversed.
In this embodiment, the predefined second target attribute values are multiple, and before it is determined that the target network traffic has the target abnormal state, the multiple second target attribute values associated with the abnormal state of the target type are traversed, that is, the attribute values of the abnormal classification are traversed. When a second target attribute value is traversed currently, the binary value of the first target attribute value and the binary value of the second target attribute value are subjected to logic AND processing to obtain a target processing result, and the first target attribute value is matched in a binary mode, so that the speed of matching the first target attribute value can be increased.
After the target processing result is obtained, it is determined whether the target processing result is greater than a target value, for example, the target value is 0, and it is determined whether the target processing result is greater than 0. And if the target processing result is judged to be larger than the target value, determining that the first target attribute value is successfully matched with the traversed second target attribute value, determining that the target network flow has an abnormal state of the target type, namely, obtaining an abnormal classification result meeting the condition, and outputting the abnormal classification result.
Optionally, when the target processing result is not greater than the target value and the plurality of second target attribute values are not traversed, determining a next second target attribute value of the plurality of second target attribute values as a currently traversed second target attribute value, and continuing to match the first target attribute value with the next second target attribute value until the plurality of second target attribute values associated with the abnormal state of the target type are traversed.
As an optional example, before determining that the target abnormal state exists in the target network traffic at step S108, the method further includes: traversing a plurality of second target attribute values associated with the first type of abnormal state, and performing logical and processing on binary values of the first target attribute values and binary values of the traversed second target attribute values to obtain a target processing result, wherein the target type includes a first type, for example, a frequency abnormal type; when the target processing result is larger than the target value, determining that the first target attribute value is successfully matched with the traversed second target attribute value; determining that the target network traffic has the abnormal state of the target type comprises: it is determined that a first type of abnormal condition exists for the target network traffic.
As an optional example, after logically and-processing the first target attribute value and the traversed target attribute value to obtain a target processing result, the method further includes: when the target processing result is not larger than the target value and a plurality of second target attribute values associated with the first type of abnormal state are not traversed, performing logic and processing on the binary number value of the first target attribute value and the binary number value of the traversed next second target attribute value to obtain a second processing result; when the second processing result is larger than the target value, determining that the first target attribute value is successfully matched with the traversed next second target attribute value; optionally, after it is determined that the first target attribute value is successfully matched with the traversed next second target attribute value, if the plurality of second target attribute values associated with the abnormal state of the first type are not traversed, continuing to match the first target attribute value according to the above method until the plurality of second target attribute values associated with the abnormal state of the target type are traversed; optionally, when the second processing result is not greater than the target value and the second target attribute values associated with the abnormal state of the first type are not traversed, continuing to match the first target attribute value according to the above method until the second target attribute values associated with the abnormal state of the target type are traversed.
As an alternative example, before logically and-processing the binary value of the first target attribute value and the binary value of the traversed next second target attribute value to obtain the second processing result, the method further includes: traversing a plurality of second target attribute values associated with the abnormal state of the second type under the condition that the plurality of second target attribute values associated with the abnormal state of the first type are traversed, and performing logic and processing on binary values of the first target attribute values and binary values of the traversed second target attribute values to obtain a third processing result, wherein the target type comprises the second type, for example, the second type is a crawler problem; when the third processing result is larger than the target value, determining that the first target attribute value is successfully matched with the traversed second target attribute value; determining that the target network traffic has the abnormal state of the target type comprises: determining that the target network traffic has a second type of abnormal condition.
As an optional implementation manner, after synthesizing at least one target attribute value associated with the same field into the first target attribute value in step S106, the method further includes: storing the first target attribute value in a target log.
In this embodiment, after at least one target attribute value associated with the same field is synthesized into a first target attribute value, that is, after the synthesized attribute value of the target log is obtained, the first target attribute value is stored in the target log, that is, the first target attribute value may be used as an attribute value finally stored in the target log, thereby saving the storage space.
As an optional implementation manner, in step S104, acquiring the set of attribute values of the target log includes: acquiring a plurality of target fields in a target log; a set including attribute values respectively associated with the plurality of target fields is determined as a set of attribute values.
In this embodiment, when acquiring the set of attribute values of the target log, part of the target fields may be randomly acquired in the target log without acquiring all the fields. Each target field may include a plurality of attribute values, and a combination of the plurality of attribute values is determined as a set of attribute values.
In the embodiment, the website generates the log under the flow to be detected, the log comprises a plurality of fields, each field can have a plurality of attribute values, different attribute values correspond to different data representations of the fields, the embodiment can utilize the binary system to identify the attribute value of each field, perform 'OR' operation on different attribute values of the same field, generate a new attribute value to store, and therefore the space for storing the attribute value is saved. In addition, the classification definition conditions of the embodiment can be freely combined, and any field can be selected from all fields in the log to be used for judging whether the target network traffic is abnormal, so that the efficiency of performing abnormal classification detection on the target network traffic is improved.
Example 2
The technical solution of the present invention is illustrated below with reference to preferred embodiments.
In this embodiment, the monitoring log collected by the third party company may include a plurality of fields, for example, including a cookie, an ip, a timestamp, a user agent (user agent), a website arrival (referrer), and the like. The network traffic may include abnormal traffic, which may be invalid traffic corresponding to a traffic rush, a crawler, an access being too frequent, and the like.
In this embodiment, the field exception first needs to be defined. For example, the field a is used for indicating a rule corresponding to the cookie, the field B is used for indicating a rule corresponding to the IP, and the field C is used for indicating a rule corresponding to the user agent. Table 1 is a definition table of field attribute exceptions according to an embodiment of the present invention.
Table 1 definition table of field attribute exceptions
| Name of field | Attribute value | Means for |
| A | 1 | cookie format exception |
| A | 2 | cookie exposure anomaly |
| A | 4 | cookie click exception |
| A | 8 | cookie changing too quickly |
| A | 16 | cookie time timeout |
| B | 1 | Reptile IP |
| B | 2 | Data center IP |
| B | 4 | Proxy IP |
| B | 8 | Standby IP |
| B | 16 | Too fast an IP change |
| B | 32 | Retention |
| C | 1 | Simple reptile UserAgents |
| C | 2 | Useragent is too short |
| C | 4 | Advanced reptile UserAgents |
| …… |
Table 2 is a definition table of an abnormality classification according to an embodiment of the present invention.
Table 2 definition table of abnormality classification
Table 3 is a storage table of the exception classification of a log according to an embodiment of the present invention.
TABLE 3 storage table for exception classification of logs
In this embodiment, the value of the attribute set is an intermediate result in the process of detecting the network traffic anomaly classification, which means that the log is subjected to matching calculation with all rules, and a: 1, A:8, A: 16, C:1 four rules; the composite attribute value in the log is that A: 1, A:8, A: 16, C:1, the corresponding attribute value is obtained by performing binary OR operation on the attribute value.
And (3) decomposing and matching the abnormal classification, namely matching the synthetic attribute value in the log with the rule definition value in the table 2, and outputting the abnormal classification results meeting the conditions.
For example, the composite attribute value of log xxx _001 is a: 25, C:1, it will simultaneously match to a:8, and C in the crawler problem: 1, namely, the traffic corresponding to the log xxx _001 has frequent anomalies and crawler problems.
Table 4 is a table of the exception classification result of the log according to the embodiment of the present invention.
Table 4 log exception classification results table
The classification results can be used in the final reporting statistics of network traffic.
FIG. 2 is a flow chart of a method of exception classification storage according to an embodiment of the present invention. As shown in fig. 2, the method comprises the steps of:
step S201, predefining attribute values of the fields under abnormal conditions.
Step S202, obtaining the attribute value of the field in the log, and matching the attribute value with the attribute value of the predefined field.
Step S203, the matched attribute values under the same field in the log are synthesized.
Step S204, recording the synthesized value into the synthesized attribute value of the log.
FIG. 3 is a flow chart of a decomposition matching method for anomaly classification according to an embodiment of the present invention. As shown in fig. 3, the method comprises the steps of:
in step S301, a plurality of attribute values of the abnormality classification are defined.
Step S302, traversing a plurality of attribute values of the abnormal classification.
Step S303, the binary value of the traversed attribute value and the binary value of the synthesized attribute value in the log are subjected to logic AND processing to obtain a processing result.
In step S304, it is determined whether the processing result is greater than 0.
After determining whether the processing result is greater than 0, if it is determined that the processing result is greater than 0, performing step S305; if it is determined that the processing result is not greater than 0, step S302 is performed.
In step S305, a qualified exception classification result is obtained, and the exception classification result is output.
Step S306, judging whether all the attribute values under the abnormal classification are traversed or not.
And after the abnormal classification result is output, judging whether all the attribute values under the abnormal classification are traversed or not. If all the attribute values under the abnormal classification are judged to be traversed or not, the matching process of the synthesized attribute values and the plurality of attribute values of the abnormal classification is finished; if all the attribute values under the abnormal classification are judged not to be traversed, step S302 is executed.
In this embodiment, each abnormal classification result may match a plurality of conditions defined by the classification, and as long as the flow to be detected satisfies one of the conditions defined by the classification, it may be determined that the abnormal condition of the flow to be detected belongs to the abnormal classification corresponding to the condition defined by the classification; the classification definition conditions of the embodiment can be freely combined, and any field can be selected from all fields in the log to be used for judging whether the flow is abnormal or not; in the embodiment, the website generates the log under the flow to be detected, the log comprises a plurality of fields, each field can have a plurality of attribute values, different attribute values correspond to different data representations of the fields, the embodiment can fully utilize the binary system to identify each field, carry out 'OR' operation on different attribute values of the same field, generate a new attribute value to be stored, and further save the space for storing the attribute value; in this embodiment, one log may simultaneously satisfy the conditions of different anomaly classification definitions, and the conditions of different anomaly classification definitions are used for calculating different anomaly classification results of the log, so that the efficiency of performing anomaly classification detection on target network traffic is improved.
Example 3
The embodiment of the invention also provides a device for detecting the abnormal classification of the network flow. It should be noted that the device for detecting abnormality classification of network traffic according to this embodiment may be used to execute the method for detecting abnormality classification of network traffic according to the embodiment of the present invention.
Fig. 4 is a schematic diagram of an apparatus for detecting abnormal classification of network traffic according to an embodiment of the present invention. As shown in fig. 4, the apparatus includes: a first acquisition unit 10, a second acquisition unit 20, a synthesis unit 30 and a first determination unit 40.
The first obtaining unit 10 is configured to obtain a target log generated by a target website at a target network traffic.
A second obtaining unit 20, configured to obtain an attribute value set of the target log, where an attribute value in the attribute value set is used to indicate a state of a field associated with the attribute value under the target attribute.
A composition unit 30 for composing the at least one target attribute value associated with the same field into a first target attribute value in case the set of attribute values comprises a predefined target attribute value.
And a first determining unit 40, configured to determine that the target network traffic has an abnormal state of the target type if the first target attribute value is successfully matched with a predefined second target attribute value, where the second target attribute value is used to indicate that a field associated with the second target attribute value belongs to the abnormal state of the target type.
Optionally, the synthesis unit comprises: the device comprises a processing module and a determining module. The processing module is used for carrying out logic OR processing on binary values of a plurality of target attribute values associated with the same field under the condition that the target attribute values associated with the same field are multiple to obtain a first target attribute value; a determining module, configured to determine the target attribute value as a first target attribute value if there is one target attribute value associated with the same field.
Optionally, the apparatus of this embodiment further comprises: the device comprises a first traversal unit, a processing unit, a second determination unit and a second traversal unit. The first traversal unit is used for traversing a plurality of second target attribute values associated with the abnormal state of the target type before determining that the target network traffic has the target abnormal state; the processing unit is used for performing logic AND processing on the binary value of the first target attribute value and the binary value of a second target attribute value traversed currently to obtain a target processing result; the second determining unit is used for determining that the first target attribute value is successfully matched with the traversed second target attribute value under the condition that the target processing result is larger than the target value; and the second traversal unit is used for determining the next second target attribute value in the plurality of second target attribute values as the currently traversed second target attribute value under the condition that the target processing result is not larger than the target value and the plurality of second target attribute values are not traversed completely.
The embodiment obtains a target log generated by a target website under target network traffic through a first obtaining unit 10, obtains an attribute value set of the target log through a second obtaining unit 20, wherein an attribute value in the attribute value set is used for indicating a state of a field associated with the attribute value under a target attribute, synthesizes at least one target attribute value associated with the same field into a first target attribute value through a synthesizing unit 30 under the condition that the attribute value set comprises a predefined target attribute value, and determines that the target network traffic has an abnormal state of a target type through a determining unit 40 under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, wherein the second target attribute value is used for indicating an abnormal state that the field associated with the second target attribute value belongs to the target type, thereby solving the technical problem of low detection efficiency of the website abnormal traffic, and further, the technical effect of improving the efficiency of classifying and detecting the abnormal traffic of the website is achieved.
Example 4
The embodiment of the invention also provides a storage medium. The storage medium comprises a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the method for detecting the abnormal classification of the network traffic.
Example 5
The embodiment of the invention also provides a processor. The processor is used for running a program, wherein the program executes the method for detecting the abnormal classification of the network traffic in the embodiment of the invention when running.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A method for detecting abnormal classification of network traffic is characterized by comprising the following steps:
acquiring a target log generated by a target website under target network traffic;
acquiring an attribute value set of the target log, wherein the attribute value set of the target log comprises attribute values obtained by calculating fields of the target log, the attribute values are used for indicating states of the fields associated with the attribute values under a target attribute, the states comprise a normal state and an abnormal state, each field corresponds to a plurality of attribute values, and different attribute values correspond to different data representations of the fields;
synthesizing at least one of the target attribute values associated with the same field into a first target attribute value if the set of attribute values includes a predefined target attribute value;
determining that the target network traffic has the abnormal state of a target type under the condition that the first target attribute value is successfully matched with a predefined second target attribute value, wherein the second target attribute value is used for indicating that a field associated with the second target attribute value belongs to the abnormal state of the target type; wherein the successful matching of the first target attribute value and the predefined second target attribute value comprises: traversing a plurality of the second target attribute values associated with an exception state of the target type; performing logical AND processing on the binary value of the first target attribute value and the binary value of the currently traversed second target attribute value to obtain a target processing result; when the target processing result is larger than a target value, determining that the first target attribute value is successfully matched with the traversed second target attribute value; and determining the next second target attribute value in the plurality of second target attribute values as the currently traversed second target attribute value under the condition that the target processing result is not larger than the target value and the plurality of second target attribute values are not traversed completely.
2. The method of claim 1, wherein synthesizing at least one of the target property values associated with the same field into a first target property value comprises:
when the target attribute values associated with the same field are multiple, performing logic or processing on binary values of the multiple target attribute values associated with the same field to obtain the first target attribute value;
determining the target attribute value as the first target attribute value if the target attribute values associated with the same field are one.
3. The method according to any of claims 1-2, wherein after synthesizing at least one of the target property values associated with the same field into a first target property value, the method further comprises:
storing the first target attribute value in the target log.
4. The method of any of claims 1-2, wherein obtaining the set of attribute values for the target log comprises:
acquiring a plurality of target fields in the target log;
determining a set comprising attribute values respectively associated with the plurality of target fields as the set of attribute values.
5. An apparatus for detecting an abnormality in classification of network traffic, comprising:
the first acquisition unit is used for acquiring a target log generated by a target website under the target network flow;
a second obtaining unit, configured to obtain an attribute value set of the target log, where an attribute value of the target log includes an attribute value obtained by calculating a field of the target log, where the attribute value is used to indicate a state of a field associated with the attribute value under a target attribute, where the state includes a normal state and an abnormal state, each field corresponds to multiple attribute values, and different attribute values correspond to different data representations of the field;
a composition unit for composing at least one of the target attribute values associated with the same field into a first target attribute value if the set of attribute values comprises a predefined target attribute value;
a first determining unit, configured to determine that the target network traffic has the abnormal state of a target type if matching between the first target attribute value and a predefined second target attribute value is successful, where the second target attribute value is used to indicate that a field associated with the second target attribute value belongs to the abnormal state of the target type, where the first determining unit further includes: a first traversal unit, configured to traverse a plurality of second target attribute values associated with an abnormal state of the target type before determining that a target abnormal state exists in the target network traffic; the processing unit is used for performing logic and processing on the binary value of the first target attribute value and the binary value of the currently traversed second target attribute value to obtain a target processing result; a second determining unit, configured to determine that the first target attribute value is successfully matched with the traversed second target attribute value when the target processing result is greater than a target value; and the second traversal unit is used for determining the next second target attribute value in the second target attribute values as the currently traversed second target attribute value under the condition that the target processing result is not larger than the target value and the second target attribute values are not traversed completely.
6. The apparatus of claim 5, wherein the synthesis unit comprises:
a processing module, configured to perform logical or processing on binary values of a plurality of target attribute values associated with the same field to obtain the first target attribute value when the plurality of target attribute values associated with the same field are multiple;
a determining module for determining the target attribute value as the first target attribute value if the target attribute value associated with the same field is one.
7. A storage medium, comprising a stored program, wherein the program, when executed, controls an apparatus in which the storage medium is located to perform the method of any one of claims 1 to 4.
8. A processor, characterized in that the processor is configured to run a program, wherein the program when running performs the method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910217643.4A CN110138720B (en) | 2019-03-21 | 2019-03-21 | Method and device for detecting abnormal classification of network traffic, storage medium and processor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910217643.4A CN110138720B (en) | 2019-03-21 | 2019-03-21 | Method and device for detecting abnormal classification of network traffic, storage medium and processor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110138720A CN110138720A (en) | 2019-08-16 |
| CN110138720B true CN110138720B (en) | 2021-08-24 |
Family
ID=67568536
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910217643.4A Active CN110138720B (en) | 2019-03-21 | 2019-03-21 | Method and device for detecting abnormal classification of network traffic, storage medium and processor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110138720B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110545292B (en) * | 2019-09-29 | 2021-07-30 | 秒针信息技术有限公司 | Abnormal flow monitoring method and device |
| CN111538704B (en) * | 2020-03-26 | 2023-09-15 | 平安科技（深圳）有限公司 | Log optimization method, device, equipment and readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102916991A (en) * | 2011-08-03 | 2013-02-06 | 中国移动通信集团公司 | Method, system and device for transmitting data |
| CN105471670A (en) * | 2014-09-11 | 2016-04-06 | 中兴通讯股份有限公司 | Flow data classification method and device |
| CN107071084A (en) * | 2017-04-01 | 2017-08-18 | 北京神州绿盟信息安全科技股份有限公司 | A kind of DNS evaluation method and device |
| KR20170106833A (en) * | 2016-03-14 | 2017-09-22 | 국방과학연구소 | A system for detecting of network anomaly and operation method thereof |
| CN107508809A (en) * | 2017-08-17 | 2017-12-22 | 腾讯科技（深圳）有限公司 | Identify the method and device of website type |
| CN107547490A (en) * | 2016-06-29 | 2018-01-05 | 阿里巴巴集团控股有限公司 | A kind of scanner recognition method, apparatus and system |
-
2019
- 2019-03-21 CN CN201910217643.4A patent/CN110138720B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102916991A (en) * | 2011-08-03 | 2013-02-06 | 中国移动通信集团公司 | Method, system and device for transmitting data |
| CN105471670A (en) * | 2014-09-11 | 2016-04-06 | 中兴通讯股份有限公司 | Flow data classification method and device |
| KR20170106833A (en) * | 2016-03-14 | 2017-09-22 | 국방과학연구소 | A system for detecting of network anomaly and operation method thereof |
| CN107547490A (en) * | 2016-06-29 | 2018-01-05 | 阿里巴巴集团控股有限公司 | A kind of scanner recognition method, apparatus and system |
| CN107071084A (en) * | 2017-04-01 | 2017-08-18 | 北京神州绿盟信息安全科技股份有限公司 | A kind of DNS evaluation method and device |
| CN107508809A (en) * | 2017-08-17 | 2017-12-22 | 腾讯科技（深圳）有限公司 | Identify the method and device of website type |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110138720A (en) | 2019-08-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106940679B (en) | Data processing method and device | |
| US11005730B2 (en) | System, method, and apparatus for high throughput ingestion for streaming telemetry data for network performance management | |
| CN110928718B (en) | Abnormality processing method, system, terminal and medium based on association analysis | |
| JP6706321B2 (en) | Method and device for service call information processing | |
| Yang et al. | A time efficient approach for detecting errors in big sensor data on cloud | |
| US20170185468A1 (en) | Creating A Correlation Rule Defining A Relationship Between Event Types | |
| CN111614690B (en) | Abnormal behavior detection method and device | |
| US10171335B2 (en) | Analysis of site speed performance anomalies caused by server-side issues | |
| US10225165B2 (en) | Apparatus and method for processing data streams in a communication network | |
| US11431792B2 (en) | Determining contextual information for alerts | |
| US11144376B2 (en) | Veto-based model for measuring product health | |
| CN112307057A (en) | Data processing method and device, electronic equipment and computer storage medium | |
| US20210136120A1 (en) | Universal computing asset registry | |
| CN110932933B (en) | Network condition monitoring method, computing device and computer storage medium | |
| CN110138720B (en) | Method and device for detecting abnormal classification of network traffic, storage medium and processor | |
| CN111176202A (en) | Safety management method, device, terminal equipment and medium for industrial control network | |
| CN111258798A (en) | Fault positioning method and device for monitoring data, computer equipment and storage medium | |
| CN104866296A (en) | Data processing method and device | |
| US10262133B1 (en) | System and method for contextually analyzing potential cyber security threats | |
| US10637878B2 (en) | Multi-dimensional data samples representing anomalous entities | |
| Foidl et al. | An approach for assessing industrial IoT data sources to determine their data trustworthiness | |
| CN116578911A (en) | Data processing method, device, electronic equipment and computer storage medium | |
| CN114969187A (en) | Data analysis system and method | |
| CN114860432A (en) | Method and device for determining information of memory fault | |
| CN107566187B (en) | SLA violation monitoring method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |