CN117220981A - Massive blacklist matching method based on AC automaton algorithm - Google Patents

Abstract

The application provides a massive blacklist matching method based on an AC automaton algorithm, and relates to the field of computer network security. The method comprises the steps of extracting character sets, groups and IDs of pattern strings in a blacklist, removing duplication, and constructing a state transition expression corresponding to each pattern string. And then, constructing each state transition expression AC automaton by using an AC automatic algorithm, combining the AC automaton by combining the states of the root node, the connecting root node and each AC automaton, and establishing a corresponding state transition table and a matching action table. And matching the data to be detected according to the state conversion table and the matching action table. And if the matching is successful, performing matching action response. Compared with the traditional blacklist matching method, the method provided by the application has higher matching speed and matching efficiency.

Description

Massive blacklist matching method based on AC automaton algorithm

Technical Field

The application relates to the field of computer network security, in particular to a massive blacklist matching method based on an AC automaton algorithm.

Background

In the field of network security, detection technology based on a blacklist is widely applied, but with the increase of the blacklist and the quantity to be detected, how to ensure the real-time performance and the efficiency of detection becomes a problem. Existing BF algorithms (Brute-Force algorithm), KMP algorithms (bitmap algorithm), BM algorithms (Boyer-Moore string search algorithm) and the like can meet the detection requirement of a blacklist to a certain extent, but still have a plurality of defects.

The Brute-Force algorithm compares character by character at each location, requiring traversal of all characters in the text and blacklist. This approach results in higher time complexity of the algorithm, which increases linearly with increasing text length and number of blacklists.

The KMP algorithm builds the longest common prefix table by preprocessing and uses this information for skip matching. Although the matching efficiency is improved under KMP algorithms, in some cases multiple backtracking and shifting in text is still required. Particularly when there are a large number of repeated characters, more backtracking operations may be required, resulting in a decrease in matching efficiency.

The BM algorithm combines the good suffix rule and the bad character rule, and builds a good suffix table and a bad character table through preprocessing, so as to perform jump matching. Although BM algorithms can quickly find matches in most cases, in certain specific cases multiple backtracking and movement may be required, especially if there are a large number of duplicate characters and the match fails, resulting in reduced matching efficiency.

Disclosure of Invention

Aiming at the problems existing in the prior art, the application aims to provide a massive blacklist matching method based on an AC automaton algorithm. The method builds an AC automaton (Aho-Corasick automaton) based on an AC automaton algorithm, and aims to improve matching speed and efficiency of matching a blacklist, and specifically comprises the following steps:

extracting a character set, a group and an ID of a pattern string in a blacklist, and removing duplication;

constructing a state transition expression corresponding to each mode string; the state transition expression is a graph structure comprising nodes and edges; wherein a node represents a state and an edge represents a transition from one state to another; the state includes at least a state name and a timestamp;

constructing an AC automaton corresponding to each state transition expression by using an AC automaton algorithm;

renaming the state of each AC automaton by using hash calculation and combining the state name and the time stamp, and merging the root nodes;

connecting the state of each AC automaton to a root node, and updating the state transition relation to obtain a state transition table and a matching action table;

according to the state transition table and the matching action table, performing multi-core and/or multi-thread matching on the data to be detected;

and after the matching is successful, performing matching action response.

Preferably, when extracting the character set, the group and the ID of the pattern string in the blacklist, the method further comprises: short pattern strings in the pattern string are encoded as bit patterns. The short pattern string is a pattern string having a character length smaller than a preset length.

Preferably, the transfer relationship between the updated states includes: updating state transition relations in each AC automaton to be connected to the combined root node or other connected state relations including normal state transition relations, epsilon transition relations, failure states, starting states and receiving states, jump and rollback relations; if the state transition relation in the AC automaton is an acceptance state, adding state information into the combined AC automaton; if the transfer conflict occurs between the states, the most specific mode strings are selected for matching or merging.

Further, the nodes and boundaries are dynamically added by using a borderless extension algorithm while updating the state transition relationship to obtain a state transition table and a matching action table.

Further, the State transition relation is updated, and the State creation is performed by using a spark State machine State compression technology while the State transition table and the matching action table are obtained.

Preferably, in performing the matching, one or more of SIMD instruction set acceleration (Single Instruction Multiple Data, single instruction stream multiple data stream instruction acceleration), longest match optimization, priority matching, and fast fail optimization are employed for optimization matching.

Preferably, the data structures employed in performing the matching include finite state automata, dictionary trees, and bitmaps.

As described above, in this technical solution, compared with the BF algorithm, KMP algorithm and BM algorithm in the prior art, since parsing, standardization and deduplication are performed on the pattern strings of the blacklist, the short pattern strings are encoded into bit patterns, and the AC automaton is constructed based on the state transition expression of each pattern string, and the method of optimizing and merging each AC automaton into a single AC automaton is utilized, the number and length of processing objects are reduced, and the logical relationship between data is reconstructed, so that the subsequent matching can be performed quickly and efficiently. In addition, as is well known, the performance of multi-core processing and multi-thread processing is obviously better than that of single-core processing and single-thread processing, and the efficiency of data processing can be greatly improved.

On the basis of the beneficial effects, the further scheme or the preferred scheme of the application also has the following beneficial effects: the state in the boundary extension algorithm can extend across the character boundary to match a plurality of characters, so that one state can simultaneously respond to a plurality of mode strings, and the matching efficiency is improved; the State compression technology of the spark State machine can reduce the number of states, improve the matching performance and reduce the memory occupation; the SIMD instruction set acceleration, the longest matching optimization, the priority matching and the quick failure optimization are all effective optimization strategies for improving the matching efficiency; and the finite state automata, the dictionary tree and the bitmap can support quick matching, so that the matching efficiency is improved.

Drawings

FIG. 1 is a schematic flow chart of one embodiment of the present application.

FIG. 2 is a schematic flow diagram of an embodiment of the present application for constructing an AC automaton.

FIG. 3 is a flow diagram of a run-time matching process according to one embodiment of the application.

Detailed Description

Aiming at the technical problems in the prior art, the application provides a massive blacklist matching method based on an AC automaton algorithm. Specifically, with the increase of blacklists and the number to be detected, how to ensure the real-time performance and efficiency of detection becomes a problem, and existing BF algorithms, KMP algorithms, BM algorithms and the like can meet the detection requirements of the blacklists to a certain extent. The BF algorithm is the simplest and direct method, but has lower performance when processing a large amount of texts and complex blacklists, and the KMP algorithm and the BM algorithm can improve the matching efficiency to a certain extent by utilizing some preprocessing skills, but the performance problems of the algorithms are gradually highlighted along with the large increase of the blacklists and the quantity to be detected.

In general, blacklist matching will be used for the following scenario.

Intrusion detection system (intrusion detection system, IDS)/intrusion prevention system (Intrusion Prevention System, IPS): the blacklist may be used to identify and block known malicious IP addresses that may be associated with malicious activity (e.g., attacks, malware propagation, etc.), and the IDS/IPs may detect and block in real-time based on the blacklist, improving network security.

Firewall rules: the network firewall may use the blacklisted address list to configure firewall rules to block inbound or outbound traffic originating from these IP addresses, which may help prevent attacks or untrusted traffic from known malicious sources.

Mail filtering: the blacklist may be used in an email filtering system to identify and intercept spam, malicious links, or malicious attachments from these addresses, which helps reduce the spread of spam and malware;

access control list (Access Control Lists, ACL): network devices (e.g., routers, switches) may use the blacklisted address list to configure ACLs, restricting access to network resources or services from these IP addresses.

Website access control: the website administrator may use the blacklist to restrict access from these IP addresses to protect the website from threat from malicious users or malicious behavior.

With the rapid development of computer technology, the information transfer rate is also increasing. On one hand, the information exchange becomes more convenient and rapid, and on the other hand, the network attack means is richer and the attack rate is more rapid. Therefore, it is an urgent real demand and responsibility of those skilled in the art to develop a network security product and method to build a network security system that responds rapidly.

Based on the time background and the actual demand, the application provides a massive blacklist matching method based on an AC automaton algorithm, which is based on the AC automaton, and particularly comprises two parts of a compiling period and a running period as shown in figure 1; the compiling period comprises the following steps:

preparing a large amount of IP blacklist data (Pattern) and IP data (Input) to be detected;

with patterns as input, each IP blacklist data in a Pattern is a Pattern string, and all IP blacklist data is a set of Pattern strings. First, a set of pattern strings provided are parsed and normalized, and their character sets, groupings, and unique IDs are extracted. Short pattern strings of shorter length in the pattern set are pre-encoded into a special pattern string-bit pattern. Specifically, a pattern string with a short length is regarded as a short pattern string, typically a fixed character string with a preset length threshold value between 4 and 8 characters, wherein no wild cards or regular expressions are included. Then, performing de-duplication processing on the pattern strings to prevent the same pattern strings from being contained in a pattern set constructed later for multiple times so as to reduce unnecessary matching attempts; after parsing and deduplication of the pattern strings, a state transition expression is built for each pattern string, bit pattern, based on the different character sets in the blacklist, which are distinguished by the groupings and IDs, the state transition expression being a graph structure containing nodes, edges, where nodes represent states and edges represent transitions from one state to another. This process is called a preprocessing and conversion operation of patterns, and state transition expressions of a plurality of pattern strings after the preprocessing and conversion operation constitute a pattern set.

As shown in fig. 2, based on the pattern set, multi-pattern string merging is performed to merge a plurality of state transition expressions into one single AC automaton. An AC automaton is a directed tagged multi-way tree structure that can match multiple patterns simultaneously in input data and accomplish the matching with linear time complexity.

Firstly, an AC automaton algorithm is used for constructing an AC automaton corresponding to each state transition expression, and then the states of the AC automaton are renamed. To prevent conflicts between states, i.e. duplication of state names, the state of each AC automaton is renamed to ensure that they remain unique after merging, a new state name is obtained by HASH operation combining the original state name and the timestamp.

The root nodes are then merged, merging the root nodes (i.e., the start states) of all AC automata into one new root node, which will serve as the root of the merged AC automata. And finally, connecting the states of all the AC automata to the new root node, and updating the transition relation among the states.

Wherein updating the transition relationship between states is a critical step that ensures that the merged AC automation can correctly match multiple pattern strings, and mainly comprises the steps of: the state transition relationships in the original AC automaton are updated to the relationships that connect to the new root node or other connected state, i.e., the state will respond to the input character in a new manner. If the state transition relation in the original AC automaton is an acceptance state representing the end of a pattern string, information of the acceptance state is required to be added to the combined AC automaton so as to correctly track the end of the pattern string when matching; when multiple AC automata merge, transition conflicts between states may occur, i.e., multiple states all attempt to respond to the same input, at which time the most specific pattern string needs to be selected to match or merge multiple transitions.

The other connected states have the following relationships: when multiple AC automata are merged, these initial states are connected to the merged root node to form one large joint state machine. In a joint state machine, other connected states refer to those states that are directly or indirectly connected to the root node, which states have the same behavior in the combined state machine and can be considered as part of the connected states.

The purpose of updating state transition relationships to connect to the merged root node or other connected state is to optimize the performance and memory usage of the state machine. The complexity of the state machine can be reduced and higher efficiency is provided in the search and matching process.

Specifically, the relationships of other connected states include:

normal state transition relationship: a normal state transition, representing a transition from one state to another, is typically triggered by a specific input character. In an AC automaton, these state transition relationships are used to construct a directed graph of matching patterns to determine whether a string matches any pattern.

Epsilon (empty) transfer relationship: epsilon transition is a special state transition that does not require input characters to trigger, is used to connect states to other states, and can be used to achieve state jumps and optimize matching performance.

Failure state: in an AC automaton there is typically a failure state, which is transitioned to when there is no matching pattern. The failed state facilitates a quick exit from the matching operation without having to check all possible states.

Start state and accept state: the start state represents the initial state of the AC automaton, which is the starting point of the matching operation. The accept state indicates that the matching operation has succeeded and that a matching pattern has been found.

Jump and rollback relationship: some AC automaton implementations may use additional skip relationships to speed up the matching operation, which may be used to quickly skip input characters or fall back to a previous state.

The connection relationship between the above states constitutes the structure of the entire consolidated state machine.

In the process of constructing the AC automaton, a borderless extension algorithm and a spark State machine State compression technology can be introduced to optimize the construction of the AC automaton.

In a conventional AC automaton, each state is responsible for matching only one character, but in a borderless extension algorithm, the state may extend across character boundaries to match multiple characters, which allows one state to respond to multiple pattern strings simultaneously. The borderless extension algorithm can process the regular expression with backtracking property with linear time complexity by utilizing borderless backtracking and matching extension characteristics in the regular expression, and introduce additional state information and pruning strategies, so that unnecessary backtracking operation is avoided.

In a conventional AC automaton, a State is created for each character, and a new State is not created for each input character immediately after the State compression technique of the spark State machine is adopted, instead, the creation of the State is performed according to the actual character requirement of the pattern strings, if multiple pattern strings share the same prefix, a new State is created only when the last character of the prefix needs to be matched, so that the creation of the State is delayed, and the creation of the State is not performed until a certain character is actually required to be matched. In other words, the sparsettate state compression technology only stores the nodes with state transition in the construction nodes, omits the nodes without state transition, and can reduce the memory occupation of the AC automaton. In addition, the technology can sort the nodes according to the occurrence frequency of the nodes in the mode set, delete the nodes with fewer occurrence times, adjust the transition edges pointing to the nodes, reconstruct the state diagram and output the state transition table.

The combined optimized AC automaton comprises two parts, a state transition table and a matching action table, wherein the state transition table is used for describing character transition relations among states in the AC automaton and determining how the AC automaton moves in Input to find matching; the matching action table is used to store matching actions associated with the states.

The operation period comprises the following steps:

and performing actual matching operation according to the specific Input by using the generated state transition table and the matching action table, and performing pattern matching with minimum expenditure. When the data to be detected, such as an input text character, arrives, the state table is searched according to the current state and the input character to determine the next state, and the data is moved character by character according to the state transition table until an acceptance state is reached, i.e. a matched pattern string is found, or the matching fails. When an accepted state is reached, a matching action table is looked up to perform the matching action associated with that state. A specific matching process is shown in fig. 3.

Preferably, in the matching stage of the running period, the matching performance is improved from the perspective of hardware by utilizing multi-core multi-threading and SIMD of the server; by utilizing the matching strategies such as longest matching optimization, priority matching, quick failure optimization and the like, the matching efficiency is improved from the software perspective.

The SIMD, i.e., a single instruction multiple data stream (Single Instruction Multiple Data) instruction set, is a hardware acceleration technique that operates on multiple data simultaneously by one instruction. In traditional computing, only one pair of data can be multiplied at a time using scalar operations, and multiple pairs of data can be processed simultaneously using SIMD acceleration. Common SIMDs have sse and avx based on the x86 architecture. Obviously, compared with the traditional single-thread processing mode, the processing efficiency of multiple data is higher.

The longest matching optimization is a matching method based on the longest matching principle, and automatically matches the longest pattern string to reduce the number of matching results and unnecessary callback triggering during matching. The method assumes that words in the text are arranged from left to right, starting from the beginning, the longest word is used as a basic processing unit, and the longest word is matched as much as possible, instead of simply cutting words or letters, thereby reducing the number of processing objects in the matching process. Therefore, if a plurality of pattern strings are matched with a part of the input text, only the longest match is reported, but all possible matches are not reported, so that the technical effects of improving the matching speed and the matching efficiency are achieved.

Priority matching allows assigning priorities to different pattern strings, and when a plurality of pattern strings match the same position, the pattern string with the highest priority will be selected as a matching result.

Fast failure optimization is used to speed up the case of a match failure, where if any pattern string cannot be matched in a certain state, it will fail fast without continuing deep matching. It can be seen that the speed and efficiency of processing data is better than that of single core or single thread, whether it is multi-core processing or multi-thread matching.

Preferably, in performing the line-by-line matching, the number structure employed includes a finite state automaton, a dictionary tree, and a bitmap. In contrast, in the conventional algorithm, an array, a pointer and a character string are mostly adopted to realize character comparison and position movement.

Finite state automata (FSM finite state machine or FSA finite state automaton) are the subjects of automaton theory, and are a computational model that is abstracted to study finite memory computational processes and certain language classes. Finite state automata possess a finite number of states, each of which can be migrated to zero or more states, and the input string determines which state to perform the migration.

The dictionary tree (Trie) is a tree structure, which is a variant of a hash tree. In the practical application of data matching, the common prefix of the character strings can be utilized to reduce the inquiry time and maximally reduce unnecessary character string comparison, thereby improving the matching efficiency.

A Bitmap (Bitmap) is a data structure widely used in indexing, data compression, and the like. Specifically, each bit is used to store a certain state, which is suitable for large-scale data, but the data state is not so much, and is usually used for judging whether a certain data exists or not. In the application, the application of the bitmap can rapidly judge whether the blacklist data exists in a certain IP data to be detected, thereby improving the matching efficiency.

Compared with BF algorithm, KMP algorithm and BM algorithm in the prior art, the method has the advantages that short mode strings are encoded into bit modes due to analysis, standardization and de-duplication processing aiming at the mode strings of the blacklist, the AC automaton is built based on a state transition expression of each mode string, the quantity and the length of processing objects are reduced by utilizing a method of optimizing and combining each AC automaton into a single AC automaton, and the logic relation among data is rebuilt, so that subsequent matching can be performed quickly and efficiently. In addition, as is well known, the performance of multi-core processing and multi-thread processing is obviously better than that of single-core processing and single-thread processing, and the efficiency of data processing can be greatly improved.

On the basis of the beneficial effects, the further scheme or the preferred scheme of the application also has the following beneficial effects: the state in the boundary extension algorithm can extend across the character boundary to match a plurality of characters, so that one state can simultaneously respond to a plurality of mode strings, and the matching efficiency is improved; the State compression technology of the spark State machine can reduce the number of states, improve the matching performance and reduce the memory occupation; the SIMD instruction set acceleration, the longest matching optimization, the priority matching and the quick failure optimization are all effective optimization strategies for improving the matching efficiency; and the finite state automata, the dictionary tree and the bitmap can support quick matching, so that the matching efficiency is improved.

The above description is only of the preferred embodiments of the present application and is not intended to limit the novel form of the present application, but various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims (7)

1. The massive blacklist matching method based on the AC automaton algorithm is characterized by comprising the following steps of:

extracting a character set, a group and an ID of a pattern string in a blacklist, and removing duplication;

constructing a state transition expression corresponding to each mode string; the state transition expression is a graph structure comprising nodes and edges; wherein a node represents a state and an edge represents a transition from one state to another; the state comprises at least a state name and a timestamp;

constructing an AC automaton corresponding to each state transition expression by using an AC automaton algorithm;

renaming the state of each AC automaton by using hash calculation and combining the state name and the time stamp, and merging the root nodes;

connecting the state of each AC automaton to a root node, and updating the state transition relation to obtain a state transition table and a matching action table;

according to the state transition table and the matching action table, performing multi-core and/or multi-thread matching on the data to be detected;

and after the matching is successful, performing matching action response.

2. The massive blacklist matching method based on the AC automaton algorithm according to claim 1, wherein the extracting the character set, the group and the ID of the pattern string in the blacklist further comprises:

the short pattern strings in the pattern string are encoded as bit patterns.

3. The method for matching a massive blacklist based on an AC automaton algorithm according to claim 1, wherein the updating the transition relation between states comprises:

updating the state transition relation in each AC automaton to be connected to the combined root node or other connected state;

the relationships of the other connected states include: normal state transfer relationship, epsilon transfer relationship, failure state, start state and accept state, jump and rollback relationship;

if the state transition relation in the AC automaton is an acceptance state, adding state information into the combined AC automaton; if the transfer conflict occurs between the states, the most specific mode strings are selected for matching or merging.

4. The method for matching a massive blacklist based on an AC automaton algorithm according to claim 1, wherein updating the state transition relation to obtain a state transition table and a matching action table further comprises:

nodes and boundaries are dynamically added using a borderless extension algorithm.

5. The method for matching a massive blacklist based on an AC automaton algorithm according to claim 1, wherein updating the state transition relation to obtain a state transition table and a matching action table further comprises:

state creation is performed using spark State machine State compression techniques.

6. The massive blacklist matching method based on the AC automaton algorithm of claim 1, wherein the matching further comprises:

one or more of SIMD instruction set acceleration, longest match optimization, priority matching, and fast fail optimization are employed for optimal matching.

7. The method for matching a massive blacklist based on an AC automaton algorithm according to claim 1, wherein the data structure adopted by the matching comprises a finite state automaton, a dictionary tree and a bitmap.