CN115296913A - Rapid arranging system suitable for flink operation rule - Google Patents
Rapid arranging system suitable for flink operation rule Download PDFInfo
- Publication number
- CN115296913A CN115296913A CN202210942264.3A CN202210942264A CN115296913A CN 115296913 A CN115296913 A CN 115296913A CN 202210942264 A CN202210942264 A CN 202210942264A CN 115296913 A CN115296913 A CN 115296913A
- Authority
- CN
- China
- Prior art keywords
- rule
- log
- component
- filtering
- flink
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001914 filtration Methods 0.000 claims abstract description 72
- 238000004458 analytical method Methods 0.000 claims abstract description 46
- 230000004044 response Effects 0.000 claims abstract description 26
- 230000002776 aggregation Effects 0.000 claims abstract description 16
- 238000004220 aggregation Methods 0.000 claims abstract description 16
- 238000012300 Sequence Analysis Methods 0.000 claims description 27
- 238000004364 calculation method Methods 0.000 claims description 11
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims description 10
- 238000012545 processing Methods 0.000 claims description 6
- 230000003044 adaptive effect Effects 0.000 claims 1
- 238000000034 method Methods 0.000 abstract description 23
- 238000001514 detection method Methods 0.000 abstract description 5
- 238000012098 association analyses Methods 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 12
- 238000011161 development Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 238000012544 monitoring process Methods 0.000 description 8
- 238000013461 design Methods 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000007405 data analysis Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000002085 persistent effect Effects 0.000 description 3
- 238000011144 upstream manufacturing Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 238000007792 addition Methods 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0245—Filtering by information in the payload
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/02—Network architectures or network communication protocols for network security for separating internal from external traffic, e.g. firewalls
- H04L63/0227—Filtering policies
- H04L63/0263—Rule management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1416—Event detection, e.g. attack signature detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
- H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic
- H04L63/1425—Traffic logging, e.g. anomaly detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/20—Network architectures or network communication protocols for network security for managing network security; network security policies in general
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a quick arranging system adapting to flink operation rules, which comprises: the log filtering component is used for acquiring log data, filtering the log data and generating a filtering log; the log association component is used for performing condition aggregation on the filtered logs to generate a converged data stream; the log analysis component is used for analyzing data characteristics in the converged data stream; and the rule response component is used for matching the data characteristics according to the generated rule template and outputting event information. The method and the system simply and flexibly configure various types of association analysis rule scenes to adapt to various security threat events generated in the actual network environment, can customize any detection rule scene only by simply dragging configuration parameters of the basic components, can accurately and efficiently analyze the security threat scenes by utilizing flink framework characteristics, can immediately adjust the rule analysis model and modify the rule parameters and release the rule parameters for the rapidly-changing security threat characteristics, and can adapt to the latest security threat characteristics in time.
Description
Technical Field
The invention belongs to the technical field of computer security, and particularly relates to a rapid arranging system suitable for a flink operation rule.
Background
Modern network environment is complicated, and enterprise network frequently suffers various attacks, and it is very necessary to make early warning before suffering from the attack. Various rules for early warning of security threats are designed and developed successively, but the implementation of a security monitoring rule from requirement analysis, development and testing to formal online is a complex and long process. And the security threat is flexible and changeable, and the security monitoring rule which is developed with great cost in the past is not applicable to the next second. In short, the existing early warning method for security threats is complex in development process and low in flexibility of scene application. Therefore, there is a need for a fast arranging system that can adapt to flink job rules efficiently, accurately and flexibly to solve the above problems.
Disclosure of Invention
In view of the above, a need exists for a fast arranging system adapted to flink operation rules, so as to overcome the problems of complex security threat early warning development and limited applicable scenarios in the prior art.
In order to solve the above technical problem, the present invention provides a fast arranging system adapted to flink job rules, which includes a log filtering component, a log association component, a log analysis component, and a rule response component, wherein:
the log filtering component is used for acquiring log data, filtering the log data and generating a filtering log;
the log association component is used for performing condition aggregation on the filtering logs to generate a converged data stream;
the log analysis component is used for analyzing data characteristics in the converged data stream;
and the rule response component is used for matching the data characteristics according to the generated rule template and outputting event information.
Further, the log filtering component is specifically configured to set an attribute and a corresponding attribute value, and interface with a third-party system, where the third-party system provides the attribute value; the log filtering component is also specifically used for providing multiple condition control, forming multiple filtering conditions by combining the attributes and the corresponding attribute values, and filtering log data hitting the multiple filtering conditions.
Further, the log association component is specifically configured to aggregate, according to a preset aggregation condition, the filter logs that satisfy the preset aggregation condition, to form the aggregated data stream.
Further, the log analysis component includes a log statistics component, a threshold comparison component, and a sequence analysis component, wherein:
the log statistic component is used for calculating the statistic characteristics of the filtering log in a set time period;
the threshold comparison component is used for comparing the statistical characteristics to generate a comparison result;
and the sequence analysis component is used for judging whether the attack sequence and the attack times meet the attack analysis conditions in a set time period.
Further, the log statistics component is specifically configured to select a plurality of different dimensions for statistics in a set time period, and calculate a statistical characteristic of the numerical values of the filter log, where the statistical characteristic includes at least one of a total number, a sum, an average value, a maximum value, and a minimum value.
Further, the threshold comparison component includes a single log computation component and a plurality of log computation components, wherein:
the single log calculation component is used for judging whether the corresponding statistical characteristics exceed a set threshold range or not when only a single filtering log is counted;
the log calculation components are used for judging whether the corresponding statistical characteristics meet joint conditions when a plurality of filtering logs are counted, wherein the joint conditions comprise combinations of a plurality of single characteristic conditions.
Further, the sequence analysis component includes a ranking component and an attack event analysis component, wherein:
the sorting component is used for carrying out corresponding data sorting according to the event time and the processing time of the flink time window; the device is also used for prolonging the waiting time according to the flink water line configuration;
and the attack event analysis component is used for determining the sequence and the attack times of different attack events according to the flink time window and the flink water line configuration.
Further, the rule response component includes a rule selection module, a rule generation module, and a rule alert component, wherein:
the rule selection module is used for constructing a rule structure according to the selected resource parameters;
the rule generating module is used for generating rule statements in an xml format according to a rule structure and submitting the rule statements to the flink cluster operation;
and the rule alarm component is used for carrying out alarm output on the filtering log hit by the rule statement.
Further, the rule selection module is specifically configured to configure resource parameters required by the rule corresponding to the operation of the flink cluster according to a user-defined service template, and construct a corresponding rule structure; entering a rule modeling page according to the selected template, setting rule parameters on a preset rule structure, and reconstructing the rule structure;
the rule generating module is specifically configured to generate a specific rule statement in an xml format according to the arranged rule structure, each component corresponds to a node in the rule statement, and each node has a corresponding child node to point to a subsequent flow direction of the data stream, where a time range of the sequence analysis component is an individual node.
Further, the rule alert component is specifically configured to: and performing alarm output on the filter log hit by the rule statement, wherein the output information comprises at least one of an alarm name, an alarm type, an attack stage, a threat level, a certainty level, a response mode, a flow limit and a custom description.
Compared with the prior art, the invention has the beneficial effects that: by arranging the log filtering component, log data are filtered, corresponding attribute values are set according to different security threat scenes for filtering, and accurate and efficient filtering can be performed on more complex log data; by setting the log association component, the data in the time period are aggregated according to a certain condition, and the data are aggregated according to different conditions and finally aggregated into one data; by arranging a log analysis component, performing multi-aspect data analysis on the converged data stream to obtain corresponding data characteristics; by setting the rule response component, the identified threats are processed, and the safety rule generates output information to other safety strategy platforms, so that a closed-loop ecosystem from analysis to alarm to disposal is realized. In conclusion, the invention can simply and flexibly configure various types of associated analysis rule scenes to adapt to various security threat events generated in the actual network environment, can customize any detection rule scene only by simply dragging basic component configuration parameters, can accurately and efficiently analyze the security threat scene by utilizing the flink framework characteristics, and can immediately adjust the rule analysis model and modify the rule parameters and release the rule parameters for the rapidly changing security threat characteristics so as to adapt to the latest security threat characteristics in time.
Drawings
FIG. 1 is a schematic structural diagram of an embodiment of a fast layout system adapted to flink operation rules according to the present invention;
fig. 2 is a schematic diagram of a rule monitoring process according to an embodiment of the present invention.
Detailed Description
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.
In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Further, "plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the described embodiments can be combined with other embodiments.
The invention provides a fast arranging system suitable for flink operation rules, which forms analysis models of various attacks by development of a plurality of components and flexible combination of the components, can customize any detection rule scene only by simply dragging the components and configuring parameters, and provides a new idea for further improving the efficiency and flexibility of security threat warning.
Before the description of the embodiments, the related words involved are paraphrased:
flink: the core of the open source stream processing framework developed by the Apache software foundation is a distributed stream data stream engine written in Java and Scala. Flink executes arbitrary stream data programs in a data parallel and pipelined manner, and Flink's pipelined runtime system can execute batch and stream processing programs. In addition, the runtime of Flink itself supports the execution of iterative algorithms;
assembly of: the data stream processing node has a liked general name, all components are automatically rendered through a data structure customized in the yaml file, and fields are added to the corresponding component pages synchronously only by adding new fields in the yaml file, so that flexible and variable rule conditions are adapted through as little development work as possible;
configuring a form: compiling a yaml file according to a specified format, wherein the file clearly specifies ui names, form types, default values, value types and whether filling is necessary or not of the page forms;
and (3) generating a rule: arranging rules according to actual scenes, and then generating an xml file which accords with the flink operation;
a rule template: summarizing the actual security attack rules, presetting a rule arrangement basic template in advance, wherein the template specifies which components the rules contain, and arranging a rule structure in advance (no specific rule parameters are set;
newly establishing a rule: selecting a rule template (if no template meeting the current service can be customized), then setting rule parameters on a rule structure preset by the template, configuring resources required to be applied when the operation is carried out on the flink cluster, and rearranging the rule structure on a drawing page after the new establishment is successful;
rule list: and displaying details of all rules, performing rule addition, deletion, modification and check operations, and displaying the running condition of the corresponding flink job by each rule.
Based on the description of the technical nouns, the prior art is a complicated and long process from requirement analysis, development and testing to formal online for the rule development of security threat monitoring, and the application scenario and the application period of the rule development of the security threat are limited in various aspects, and the flexibility is not sufficient. Therefore, the invention aims to provide an efficient, accurate and flexible fast arranging system for adapting to the flink operation rule.
Specific examples are described in detail below:
the embodiment of the present invention provides a fast arranging system adapted to a flink operation rule, and as seen in fig. 1, fig. 1 is a schematic structural diagram of an embodiment of a fast arranging system adapted to a flink operation rule provided by the present invention, and includes a log filtering component 101, a log association component 102, a log analysis component 103, and a rule response component 104, where:
the log filtering component 101 is configured to obtain log data, filter the log data, and generate a filtered log;
the log association component 102 is configured to perform conditional aggregation on the filtered logs to generate a converged data stream;
the log analysis component 103 is configured to analyze data characteristics in the aggregated data stream;
the rule response component 104 is configured to match the data features according to the generated rule template, and output event information.
In the embodiment of the invention, the log filtering component is arranged to filter the log data, and corresponding attribute values are set according to different security threat scenes to filter the log data, so that the log data can be accurately and efficiently filtered no matter how complex the log data are; by setting the log association component, the data in the time period are aggregated according to a certain condition, and the data are aggregated according to different conditions and finally aggregated into one data; by arranging a log analysis component, performing multi-aspect data analysis on the converged data stream to obtain corresponding data characteristics; by setting the rule response component, the identified threats are processed, and the safety rule generates output information to other safety strategy platforms, so that a closed-loop ecosystem from analysis to alarm to disposal is realized.
It should be noted that, according to various attack cases which have already occurred, the invention abstracts them into six structural components, and the six structural components can be flexibly combined to form various attack analysis models. Meanwhile, in order to facilitate operation, the invention provides a fast arranging system suitable for the flink operation rule, which liberates a new safety monitoring rule from a complex development process and can customize any detection rule scene only by simply dragging components and configuring parameters. By utilizing the flink framework characteristics and the strong component capability designed by the invention, the security threat scene can be accurately and efficiently analyzed, the capability of monitoring the running condition in real time is realized aiming at the operation on the flink, the alarm can be timely sent to a specified platform or system when the security threat is found, and the sudden abnormal operation of the flink operation can be timely repaired. Meanwhile, for the rapidly changing security threat characteristics, the rule analysis model can be immediately adjusted to modify the rule parameters and release the rule parameters, so that the latest security threat characteristics can be adapted in time.
As a preferred embodiment, the log filtering component is specifically configured to set an attribute and a corresponding attribute value, and interface with a third-party system, where the third-party system provides the attribute value; the log filtering component is also specifically used for providing multiple condition controls, combining the attributes and the corresponding attribute values to form multiple filtering conditions, and filtering log data hitting the multiple filtering conditions.
In the embodiment of the invention, the log filtering component is arranged to preliminarily filter log data, and a plurality of filtering conditions are utilized to realize efficient and accurate filtering effect.
In a specific embodiment of the present invention, the log filtering component specifically implements the following functions:
all security rules can receive all log data of the equipment for analysis, the data volume is very large, and great pressure can be caused to subsequent data analysis if the log data are not filtered, so the invention designs a log filtering component, the component is used as an entrance of all the rules, all log data streams of the equipment can pass through the log filtering component, the log filtering component can show all attributes in the log data (all attributes are obtained by analyzing original logs through log generalization and are not discussed in the patent), and corresponding attribute values are set for filtering according to different security threat scenes (for example, some attacks can only be carried out from the abroad, and the invention sets the data source region to be oversea). The invention provides a plurality of ways to set attribute values, can input attribute values in a customized manner, can set fixed options of some attributes through a data dictionary (because some attribute values are slowly summarized in actual business, the data dictionary can be updated in a customized manner in real time), can also be connected with a third-party system, and can use data provided by the third-party system as the attribute values of the invention, for example, some risk ports need to be filtered, but specific risk ports are not clear, so a system special for maintaining the risk ports can be specified, and the risk ports are provided by the system. In order to more accurately filter log data, the invention also provides rich condition control, including equal to, greater than OR equal to, less than OR equal to, starting with, ending with, containing with, belonging to, AND OR association between a plurality of conditions. And each condition can also be embedded with a condition group (the condition group is an independent condition design), the design of the infinite nesting doll forms a powerful log filtering component, and accurate and efficient filtering can be performed no matter how complex log data exist.
As a preferred embodiment, the log association component is specifically configured to aggregate, according to a preset aggregation condition, the filter logs that satisfy the preset aggregation condition, so as to form the aggregated data stream.
In the embodiment of the invention, the log association component is arranged, the aggregation condition is designed in a complex way, and the complex condition aggregation is supported.
In a specific embodiment of the present invention, the functions specifically implemented by the log association component are as follows:
in the case analysis of the actual security threats, the invention finds that some threats need to filter different logs, and then aggregates data in the time period according to a certain condition within a certain time (for example, a WAF attack needs to filter a WAF log and an IPS log respectively, and then aggregates the WAF log and the IPS log according to the same destination IP). According to the characteristic, the log association component is designed, and the log association component has the function of receiving data streams output by a plurality of upstream components, then carrying out data aggregation according to different conditions, and finally aggregating the data streams into one data stream. Meanwhile, in order to ensure the high availability of the log association component, the invention carries out complex design on the aggregation condition, adds the association of AND AND OR among a plurality of conditions AND leads the association to support complex condition aggregation. Wherein the timeframe also supports flink timewindows and water line characteristics, see the relevant description within the sequence analysis for details.
As a preferred embodiment, the log analysis component comprises a log statistics component, a threshold comparison component, and a sequence analysis component, wherein:
the log statistical component is used for calculating the statistical characteristics of the filtering log in a set time period;
the threshold comparison component is used for comparing the statistical characteristics to generate a comparison result;
and the sequence analysis component is used for judging whether the attack sequence and the attack times meet the attack analysis conditions in a set time period.
In the embodiment of the invention, a plurality of analysis components are arranged to analyze data in multiple aspects and angles.
In a preferred embodiment, the log statistics component is specifically configured to select a plurality of different dimensions for statistics in a set time period, and calculate a statistical characteristic of the numerical values of the filter log, where the statistical characteristic includes at least one of a total number, a sum, an average value, a maximum value, and a minimum value.
In the embodiment of the invention, a log counting component is arranged for carrying out multi-dimensional data statistics.
In a specific embodiment of the present invention, the specific implementation functions of the log statistics component are as follows:
how to define normal access flow and attack flow, and reducing false alarm of security threat is also a problem to be faced, therefore, a flexible component is needed to control the bealock, log statistics is responsible for critical data calculation, the log statistics receives data stream output by an upstream component, then data statistics is carried out in a set time period, a plurality of different dimensions can be simultaneously selected for carrying out statistics, and the calculation type supports count (total number), sum (summation), avg (average value), max (maximum value) and min (minimum value). Where the time ranges herein also support flink time windows and water line characteristics, see the relevant description of sequence analysis for details.
As a preferred embodiment, the threshold comparison component comprises a single log computation component and a plurality of log computation components, wherein:
the single log calculation component is used for judging whether the corresponding statistical characteristics exceed a set threshold range or not when only a single filtering log is counted;
the log calculation components are used for judging whether the corresponding statistical characteristics meet the joint conditions or not when a plurality of filtering logs are counted, wherein the joint conditions comprise the combination of a plurality of single characteristic conditions.
In the embodiment of the invention, a threshold comparison component is arranged to satisfy single-value comparison or combined comparison.
In a specific embodiment of the present invention, the threshold comparison component is implemented as follows:
the threshold comparison generally occurs together with the log statistics, and when only one log statistics is available, the threshold comparison of the invention is used for single value calculation, which is mainly to compare whether the data statistics exceeds the threshold range set by the invention or not. When two OR more log statistics exist, the threshold comparison is multi-valued comparison, changes between the two data are mainly compared (for example, whether the average value of data flow in the last day is far higher than the average value of data flow in the last week OR not is calculated), AND when the number of the connected log statistics components is more than two, the threshold condition can be associated through AND AND OR, so that the joint comparison among a plurality of log statistics can be met, AND the interval range comparison of a single-valued condition can be carried out.
As a preferred embodiment, the sequence analysis component comprises a ranking component and an attack event analysis component, wherein:
the sorting component is used for carrying out corresponding data sorting according to the event time and the processing time of the flink time window; the system is also used for prolonging the waiting time according to the flink water line configuration;
and the attack event analysis component is used for determining the sequence and the attack times of different attack events according to the flink time window and the flink water line configuration.
In the embodiment of the invention, the corresponding events are subjected to time sequencing and analysis.
In a specific embodiment of the present invention, the sequence analysis component is implemented as follows:
there are also security threats that must satisfy the order and number of attacks within a fixed time frame, for example "eternal blue" must first have a DNS log, then a TCP traffic log, and finally a WINDOWS host log. However, in an actual service scenario, under the influence of a network environment, the log data does not arrive at the sequence analysis component in an ideal state in order, for example, the DNS log is generated at 0 point 4 minutes and 5 seconds, and the TCP log is generated at 0 point 4 minutes and 7 seconds, but for a network reason, when finally arriving at the sequence analysis component for analysis, the TCP log arrives first and the DNS log arrives later. There is also a case of data delay, for example, the invention sets the permanent blue time calculation window to be 5 minutes, the DNS log is generated just at 5 minutes, but because of the network delay, it reaches the sequence analysis component at 6 minutes, and the time calculation window is already closed, so this data will be missed, for this case, the invention adds a flink water line configuration to the sequence analysis component to solve (simply, let the time window close more than one minute, so the DNS log will not be missed), and makes full use of the characteristics of the flink framework, so that the component of the invention can be adapted to various unexpected situations in actual business.
As a preferred embodiment, the rule response component comprises a rule selection module, a rule generation module and a rule alarm component, wherein:
the rule selection module is used for constructing a rule structure according to the selected resource parameters;
the rule generating module is used for generating rule statements in an xml format according to a rule structure and submitting the rule statements to the flink cluster operation;
and the rule alarm component is used for carrying out alarm output on the filtering log hit by the rule statement.
In the embodiment of the invention, a rule response component is arranged, and the generated alarm is finally sent to a specified platform according to the alarm parameters after the rule matching is successful.
As a preferred embodiment, the rule selection module is specifically configured to configure, according to a user-defined service template, resource parameters required by a rule corresponding to a job when the flink cluster runs, and construct a corresponding rule structure; and entering a rule modeling page according to the selected template, setting rule parameters on a preset rule structure, and reconstructing the rule structure.
In the embodiment of the invention, a rule selection module is arranged, and the construction of the rule is realized by utilizing the parameter setting of a user.
As a preferred embodiment, the rule generating module is specifically configured to generate a rule statement in a specific xml format according to the laid-up rule structure, each component corresponds to a node in the rule statement, and each node has a corresponding child node to point to a subsequent flow direction of the data stream, where a time range of the sequence analysis component is a single node.
In the embodiment of the invention, the constructed rule structure is utilized to form corresponding rule statements pointing to different component flow directions.
In a specific embodiment of the present invention, the specific implementation functions of the rule selection module and the rule generation module are as follows:
in practical cases, the invention finds that part of security threats can summarize characteristic rules, and the invention analyzes the rules and summarizes the rules into a rule model to form a standard structural framework, which is the template of the invention. For example, DDos attack event, which has a remarkable characteristic that the flow rate is abnormally and suddenly increased in a time period, the invention designs the components of the template, including log filtering, log statistics, threshold comparison and rule response, and connects according to the sequence. When a DDos attack rule is created, a DDos attack template is selected, log filtering conditions are input in the first step according to a connection structure, the flow peak value of a time period is counted in the second step, the peak value is compared with a set threshold value in the third step, and information and a docking platform which are output after the attack conditions are met are set in the fourth step. Thus, the safety analysis rule of the DDos attack is successfully established. The application of the rule template can reduce the time for arranging the structural framework when a new rule is established, so that the purpose of fast arrangement is achieved, and a guiding effect is played for a new user, wherein:
the specific functions of the rule selection module include: and selecting a rule template (no template which accords with the self service can be customized), and configuring resource parameters which are required when the job runs in the flink cluster and correspond to the rule. The method mainly comprises the steps that the memory and the parallelism required by operation of the operation are configured, and the system can obtain the residual allocable resources in the flink cluster before configuration, so that a user cannot exceed the residual available resources when configuring the resources of the operation, and the problem that the operation submission fails due to insufficient residual resources is solved. Further, entering a rule modeling page according to the selected template, setting rule parameters on a preset rule structure, and reconstructing the rule structure (rearranging the dragging node);
the specific functions of the rule generation module include: and generating specific rule xml according to the arranged rule modeling, wherein each component corresponds to a node in the xml, each node has a < success > child node so as to point to the subsequent flow direction of the data stream, and the time range of the sequence analysis component is a single node.
The node attributes are as follows:
id: id (unique identification) of the corresponding component;
name: corresponding to the component name;
type: the corresponding component type comprises data source (data source, dynamic acquisition in system configuration, generally not modified, so it is not designed as component), filter (log filtering), window (time range object in component), aggregation (log statistics), join (log association), operator (threshold comparison), sequence (sequence analysis), action (rule response);
a method: specifically, statements executed in the flink generate statements (basically the same as sql statement specifications) meeting the flink semantic specifications according to the configured confidence in the components, and the generation logic takes log filtering as an example as follows:
firstly, defining a template of a flink semantic specification of each component, wherein log filtering comprises a basic statement select from $ $ AND a conditional statement $ $ field condition $ $ value, AND the condition of obtaining the log filtering component is that source ip is equal to 1.1.1.1AND destination ip is equal to 2.2.2.2. The invention replaces a first condition by circularly traversing the filtering conditions to obtain srcip (replace keyword $ field) = (replace keyword condition) 1.1.1.1 (replace keyword $ value), then splices a condition association keyword AND, AND then traverses the second condition to replace dstip =2.2.2, AND finally splices the basic statement to obtain a final method with select from $ field $ word srcp = 1.1.1.1.1.and dstip =2.2.2. The method generated by other components is similar to log filtering in a way of generating method, and is also generated by an alternative method;
successoros: and the next action after the data passes through the current component, the default of the group attribute in the node as copy represents that the data stream is copied, and the node attribute indicates that the copied data stream is specifically directed to which downstream node.
As a preferred embodiment, the rule alert component is specifically configured to: and performing alarm output on the filter log hit by the rule statement, wherein the output information comprises at least one of an alarm name, an alarm type, an attack stage, a threat level, a certainty level, a response mode, a limited flow and a custom description.
In the embodiment of the invention, by setting the rule alarm component, the efficient alarm description is realized and the alarm description is intuitively presented to the user.
In a specific embodiment of the present invention, after a rule of security analysis is created, the present invention needs to process the identified threat, where a rule response is needed, which is an exit of all rules, and sets output information for the rule that satisfies the condition, where the output information includes an alarm name, an alarm type, an attack stage (telling the present invention that this alarm may make a threat action), a threat level, a certainty level (setting the reliability of this alarm), a response mode (configuring the actions that the present invention can make, which may be the own platform alarm or other third party platform), a current limit (mainly to prevent that some alarms with lower certainty level push data all the time cause excessive pressure on the platform that receives the alarm information), a custom description (automatically generating popular and understandable alarm descriptions according to the rule drawn by the present invention, for example, through XXX events and XXX events, IP statistics for a purpose within 5 minutes are greater than a predetermined threshold 1, triggering the custom alarm description can let the user have a check effect on the drawn rule, and also can realize a basic function of a XXX event, and then the present invention can output a safety analysis rule from a closed-loop analysis to other safety analysis system, which can handle the safety analysis rules;
it should be noted that, as long as the components are connected, data in the upstream component can be automatically shared in the downstream component, so as to achieve the purpose of highly automated and rapid layout. Meanwhile, abnormal conditions in actual services are solved by using the characteristics of the flink time window and the water line in sequence analysis, and the method is applied to log association as well as log statistical components. Through the six designed components, the attack scenes of most security threats can be basically covered by flexibly combining the six components, certain particularly complex attack scenes cannot be arranged into a rule structure model, and a more flexible rule importing function is further provided (see a rule importing part in logic details). Wherein, yaml is a format with high readability for expressing data serialization; XML is an extensible markup language, a subset of the standard generalized markup language. Is a markup language for marking electronic documents to be structured.
The following describes the implementation process of the generation rule in detail by taking the persistent blue as an example:
firstly, reading a data source in the configuration, correspondingly generating a regular xml data source node, wherein the permanent blue has three log filtering components, so that the data source can respectively flow to three logs for filtering, and the < success > node comprises the ids of the three log filtering nodes;
secondly, the data filtered by the three log filtering nodes are all flow direction sequence analysis nodes, so that the < success > nodes of the three log filtering nodes are all directed to the same downstream node;
thirdly, because the sequence analysis component has a time object, the data inflow firstly flows into a time node and then reaches a corresponding sequence analysis node;
fourthly, the data meeting the condition finally is transmitted to the rule response node, because the rule response is the end node, the downstream nodes are empty, and finally the security threat details are generated at the rule response node according to the detailed configuration and are sent to the specified platform according to the configuration in the method.
As a specific embodiment, the process of submitting a task specifically includes: and submitting the corresponding xml file generated by the rule to the built flink cluster for running, wherein only one job corresponding to one rule is allowed to run at the same time, and if the job is running, the job is stopped and then submitted.
In the embodiment of the present invention, it is,
as a specific embodiment, the process of stopping the task specifically includes: the stopping rules correspond to jobs running in the flink cluster.
In the embodiment of the present invention, it is,
as a specific embodiment, the process of enabling/disabling the rule specifically includes: and enabling the newly-built rules to be effective (enabled) or invalid (disabled), and if a rule has a flink job which is running, synchronously stopping and deleting the corresponding job when the rule is disabled. In the embodiment of the present invention, it is,
as a specific embodiment, the process of deleting a rule specifically includes: and deleting the rule, and synchronously stopping and deleting if the corresponding job running in the flink cluster exists. In the embodiment of the present invention, it is,
as a specific embodiment, the process of rule importing specifically includes: and providing a fixed xml template, importing the template into a system after a user fills parameters, directly generating rules reversely for modeling, and submitting the rules to a flink cluster for operation after support of rearrangement. Indeed, some particularly complex attack scenarios cannot arrange a regular structure model, and the method also supports online editing of xml and submitting the xml to a flink cluster for operation after storage. In the embodiment of the present invention, it is,
taking persistent blue as an example, and referring to fig. 2, fig. 2 is a schematic diagram of an embodiment of a rule monitoring process provided by the present invention, and details a "persistent blue lux virus" rule monitoring modeling process are as follows:
step 1, setting three log filtering components, and respectively filtering a DNS analysis log, a Windows host log and a TCP flow log;
step 2, transmitting the filtered data stream to the same sequence analysis component, and setting the sequence and the frequency of the filtered data stream in the component;
step 3, transmitting the data which is analyzed by the sequence analysis component and meets the conditions to a rule response;
and 4, setting alarm parameters after rule matching is successful in rule response, and finally sending the generated alarm to a specified platform.
The invention discloses a fast arranging system suitable for flink operation rules, which realizes the filtering of log data by arranging a log filtering component, sets corresponding attribute values according to different security threat scenes for filtering, and can accurately and efficiently filter the log data no matter how complex the log data are; by setting the log association component, the data in the time period are aggregated according to a certain condition, and the data are aggregated according to different conditions and finally aggregated into one data; by arranging a log analysis component, performing multi-aspect data analysis on the converged data stream to obtain corresponding data characteristics; by setting the rule response component, the identified threats are processed, and the safety rule generates output information to other safety strategy platforms, so that a closed-loop ecosystem from analysis to alarm to disposal is realized.
According to the technical scheme, various types of association analysis rule scenes are simply and flexibly configured to adapt to various security threat events generated in the actual network environment, any detection rule scene can be customized only by simply dragging basic component configuration parameters, the security threat scenes can be accurately and efficiently analyzed by using flink framework characteristics, and for the rapidly-changing security threat characteristics, a rule analysis model can be immediately adjusted, rule parameters can be modified and released, and the latest security threat characteristics can be adapted in time.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.
Claims (10)
1. A fast arranging system adapting to flink job rules is characterized by comprising a log filtering component, a log association component, a log analysis component and a rule response component, wherein:
the log filtering component is used for acquiring log data, filtering the log data and generating a filtering log;
the log association component is used for performing condition aggregation on the filter logs to generate a converged data stream;
the log analysis component is used for analyzing data characteristics in the converged data stream;
and the rule response component is used for matching the data characteristics according to the generated rule template and outputting event information.
2. The system for rapidly orchestrating according to flink job rules according to claim 1, wherein the log filter component is specifically configured to set attributes and corresponding attribute values, and interface with a third-party system, wherein the third-party system provides the attribute values; the log filtering component is also specifically used for providing multiple condition control, forming multiple filtering conditions by combining the attributes and the corresponding attribute values, and filtering log data hitting the multiple filtering conditions.
3. The fast orchestration system according to claim 1, wherein the log association component is specifically configured to aggregate, according to a preset aggregation condition, the filter logs that satisfy the preset aggregation condition, to form the aggregated data stream.
4. The system for rapid orchestration according to claim 1, wherein the log analysis component comprises a log statistics component, a threshold comparison component, and a sequence analysis component, wherein:
the log statistical component is used for calculating the statistical characteristics of the filtering log in a set time period;
the threshold comparison component is used for comparing the statistical characteristics to generate a comparison result;
and the sequence analysis component is used for judging whether the attack sequence and the attack times meet attack analysis conditions or not in a set time period.
5. The system for fast orchestration according to claim 4, wherein the log statistics component is specifically configured to select a plurality of different dimensions for statistics in a set time period, and calculate statistical characteristics of the values of the filter log, wherein the statistical characteristics include at least one of a total number, a sum, an average value, a maximum value, and a minimum value.
6. The system for rapid orchestration according to flink job rules according to claim 4, wherein the threshold comparison component comprises a single log computation component and a plurality of log computation components, wherein:
the single log calculation component is used for judging whether the corresponding statistical characteristics exceed a set threshold range or not when only a single filtering log is counted;
the log calculation components are used for judging whether the corresponding statistical characteristics meet the joint conditions or not when a plurality of filtering logs are counted, wherein the joint conditions comprise the combination of a plurality of single characteristic conditions.
7. The system of fast orchestration according to adaptive flink job rules of claim 4, wherein the sequence analysis component comprises a ranking component and an attack event analysis component, wherein:
the sorting component is used for carrying out corresponding data sorting according to the event time and the processing time of the flink time window; the system is also used for prolonging the waiting time according to the flink water line configuration;
and the attack event analysis component is used for determining the sequence and the attack times of different attack events according to the flink time window and the flink water line configuration.
8. The system for rapid orchestration of flink job-adapted rules according to claim 1, wherein the rule response component comprises a rule selection module, a rule generation module, and a rule alarm component, wherein:
the rule selection module is used for constructing a rule structure according to the selected resource parameters;
the rule generating module is used for generating rule statements in an xml format according to a rule structure and submitting the rule statements to the flink cluster operation;
and the rule alarm component is used for carrying out alarm output on the filtering log hit by the rule statement.
9. The system for rapidly arranging flink job rules according to claim 8, wherein the rule selection module is specifically configured to configure resource parameters required by the rule corresponding to the job when the flink cluster runs according to a user-defined service template, and construct a corresponding rule structure; entering a rule modeling page according to the selected template, setting rule parameters on a preset rule structure, and reconstructing the rule structure;
the rule generating module is specifically configured to generate a specific rule statement in an xml format according to the arranged rule structure, each component corresponds to a node in the rule statement, and each node has a corresponding child node to point to a subsequent flow direction of the data stream, where a time range of the sequence analysis component is an individual node.
10. The system of rapidly orchestrating according to flink job rules according to claim 8, wherein the rule alert component is specifically configured to: and performing alarm output on the filter log hit by the rule statement, wherein the output information comprises at least one of an alarm name, an alarm type, an attack stage, a threat level, a certainty level, a response mode, a limited flow and a custom description.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210942264.3A CN115296913A (en) | 2022-08-05 | 2022-08-05 | Rapid arranging system suitable for flink operation rule |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210942264.3A CN115296913A (en) | 2022-08-05 | 2022-08-05 | Rapid arranging system suitable for flink operation rule |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115296913A true CN115296913A (en) | 2022-11-04 |
Family
ID=83829130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210942264.3A Pending CN115296913A (en) | 2022-08-05 | 2022-08-05 | Rapid arranging system suitable for flink operation rule |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115296913A (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019210484A1 (en) * | 2018-05-03 | 2019-11-07 | Siemens Aktiengesellschaft | Analysis device, method and system for operational technology system and storage medium |
| CN111404909A (en) * | 2020-03-10 | 2020-07-10 | 上海豌豆信息技术有限公司 | Security detection system and method based on log analysis |
| CN111597550A (en) * | 2020-05-14 | 2020-08-28 | 深信服科技股份有限公司 | Log information analysis method and related device |
| CN112104633A (en) * | 2020-09-07 | 2020-12-18 | 西安电子科技大学 | Attack chain construction method based on log correlation analysis |
| CN112685394A (en) * | 2020-12-25 | 2021-04-20 | 北京鼎普科技股份有限公司 | Real-time threat information correlation method, device and system based on Flink |
| CN112738016A (en) * | 2020-11-16 | 2021-04-30 | 中国南方电网有限责任公司 | Intelligent security event correlation analysis system for threat scene |
| CN113141368A (en) * | 2021-04-27 | 2021-07-20 | 天翼电子商务有限公司 | System supporting real-time security threat association analysis of mass data |
| CN113810362A (en) * | 2021-07-28 | 2021-12-17 | 中国人寿保险股份有限公司上海数据中心 | Safety risk detection and disposal system and method thereof |
| US20220038330A1 (en) * | 2020-07-31 | 2022-02-03 | Hewlett Packard Enterprise Development Lp | Systems and methods for predictive assurance |
| CN114327678A (en) * | 2021-12-29 | 2022-04-12 | 中电福富信息科技有限公司 | Real-time data processing system and method supporting multiple engines |
-
2022
- 2022-08-05 CN CN202210942264.3A patent/CN115296913A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019210484A1 (en) * | 2018-05-03 | 2019-11-07 | Siemens Aktiengesellschaft | Analysis device, method and system for operational technology system and storage medium |
| CN111404909A (en) * | 2020-03-10 | 2020-07-10 | 上海豌豆信息技术有限公司 | Security detection system and method based on log analysis |
| CN111597550A (en) * | 2020-05-14 | 2020-08-28 | 深信服科技股份有限公司 | Log information analysis method and related device |
| US20220038330A1 (en) * | 2020-07-31 | 2022-02-03 | Hewlett Packard Enterprise Development Lp | Systems and methods for predictive assurance |
| CN112104633A (en) * | 2020-09-07 | 2020-12-18 | 西安电子科技大学 | Attack chain construction method based on log correlation analysis |
| CN112738016A (en) * | 2020-11-16 | 2021-04-30 | 中国南方电网有限责任公司 | Intelligent security event correlation analysis system for threat scene |
| CN112685394A (en) * | 2020-12-25 | 2021-04-20 | 北京鼎普科技股份有限公司 | Real-time threat information correlation method, device and system based on Flink |
| CN113141368A (en) * | 2021-04-27 | 2021-07-20 | 天翼电子商务有限公司 | System supporting real-time security threat association analysis of mass data |
| CN113810362A (en) * | 2021-07-28 | 2021-12-17 | 中国人寿保险股份有限公司上海数据中心 | Safety risk detection and disposal system and method thereof |
| CN114327678A (en) * | 2021-12-29 | 2022-04-12 | 中电福富信息科技有限公司 | Real-time data processing system and method supporting multiple engines |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10678669B2 (en) | Field content based pattern generation for heterogeneous logs | |
| US8185488B2 (en) | System and method for correlating events in a pluggable correlation architecture | |
| US8191081B2 (en) | Condition-based event filtering | |
| WO2020228276A1 (en) | Network alert method and device | |
| US8132182B2 (en) | Parallel processing of triggering rules in SIP event notification filters | |
| CN110113315A (en) | A kind of processing method and equipment of business datum | |
| CN114338746A (en) | Analysis early warning method and system for data collection of Internet of things equipment | |
| CN110764894A (en) | Timed task management method, device, equipment and storage medium | |
| CN111510446B (en) | Attack detection method and device, electronic equipment and storage medium | |
| CN104901833B (en) | A kind of method and device for the equipment that notes abnormalities | |
| EP3857853A1 (en) | System and methods for automated computer security policy generation and anomaly detection | |
| CN113098854B (en) | Task arranging method, system, storage medium and electronic equipment | |
| CN114356286A (en) | Method and system for developing low-coding interface | |
| US20240089279A1 (en) | Method and network node for detecting anomalous access behaviours | |
| CN115296913A (en) | Rapid arranging system suitable for flink operation rule | |
| WO2021244441A1 (en) | Service configuration method and apparatus therefor | |
| CN113282557A (en) | Big data log analysis method and system based on Spring framework | |
| CN101964922B (en) | Abnormal condition capturing method and device | |
| KR102426889B1 (en) | Apparatus, method and program for analyzing and processing data by log type for large-capacity event log | |
| CN115361231B (en) | Host abnormal flow detection method, system and equipment based on access baseline | |
| CN115934782B (en) | Method for analyzing and processing security log and computer storage medium | |
| CN110445793B (en) | Analysis method of analysis engine with node thread level non-redundant computation | |
| CN115809262A (en) | Alarm processing method, device and equipment | |
| CN116028451A (en) | Log analysis method and related equipment | |
| WO2024068833A1 (en) | Processing and/or generating cybersecurity telemetry data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |