WO2023245911A1

WO2023245911A1 - Complex event detection method and apparatus and storage medium

Info

Publication number: WO2023245911A1
Application number: PCT/CN2022/123107
Authority: WO
Inventors: 任超超; 陈政; 黄国财; 张慢丽
Original assignee: 深圳前海微众银行股份有限公司
Priority date: 2022-06-24
Filing date: 2022-09-30
Publication date: 2023-12-28
Also published as: CN115048146A

Abstract

A complex event detection method and apparatus, and a storage medium. The method comprises: obtaining an operation event stream, there are a plurality of events in the operation event stream (S101); binding a plurality of event rules and a plurality of pieces of event determination logic to each event in the operation event stream (S102); using a calculation engine to determine the matchability between the operation event stream and the plurality of bound event rules (S103); in the case of matching, screening, from the operation event stream, for a target event having an event operation time less than a target watermark, the target watermark being a time determined according to a plurality of event operation times corresponding to the operation event stream (S104); and determining a determination result of the target event according to the plurality of pieces of event determination logic, so as to determine the target event as a complex event according to the determination result (S105).

Description

A complex event detection method, device and storage medium

Cross-references to related applications

This application claims priority to a Chinese patent application filed with the China Patent Office on June 24, 2022, with application number 202210739018.8 and the application title "A complex event detection method, device and storage medium", the entire content of which is incorporated by reference. in this application.

Technical field

This application relates to the field of financial technology, and in particular to a complex event detection method, device, and storage medium.

Background technique

With the development of computer technology, more and more technologies are applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, Fintech also puts forward technical requirements. higher requirements. In the field of financial technology, complex event detection is becoming more and more important. Most of the existing technologies use Flink to detect complex technologies.

In the existing technology, Flink complex event detection relies on the rule broadcast stream and n predefined rule calculation units to implement complex event detection. If the new rule is not in the predefined rule calculation unit, you need to add a new rule first. The code of the calculation unit is compiled and run, thus reducing the speed when adding complex event detection rules.

Contents of the invention

In order to solve the above technical problems, embodiments of the present application are expected to provide a complex event detection method, device, and storage medium, which can increase the speed when adding complex event detection rules.

The technical solution of this application is implemented as follows:

The embodiment of the present application provides a complex event detection method. The complex event detection method includes:

Obtain an operation event stream, where the number of events in the operation event stream is multiple;

Bind multiple event rules and multiple event determination logic to each event in the operation event stream;

Using a calculation engine to determine the match between the operation event stream and the multiple bound event rules;

In the case of matching, target events whose event operation time is less than a target water level are screened from the operation event stream; the target water level is a time determined based on multiple event operation times corresponding to the operation event stream;

The determination result of the target event is determined according to the plurality of event determination logics, so that the target event is determined to be a complex event according to the determination result.

An embodiment of the present application provides a complex event detection device, which includes:

The acquisition part is configured to acquire an operation event stream, where the number of events in the operation event stream is multiple;

The binding part is configured to bind multiple event rules and multiple event determination logic to each event in the operation event stream;

The determination part is configured to use the calculation engine to determine the matching between the operation event stream and the multiple bound event rules; determine the determination result of the target event according to the multiple event determination logic, so as to determine the determination result according to the determination result. The target event is a complex event;

The filtering part is configured to, in the case of matching, filter the target events whose event operation time is less than the target water level from the operation event stream; the target water level is a plurality of event operations corresponding to the operation event stream. Time determined time.

A memory, a processor and a communication bus. The memory communicates with the processor through the communication bus. The memory stores a complex event detection program executable by the processor. When the complex event detection program is During execution, the above-mentioned complex event detection method is executed by the processor.

Embodiments of the present application provide a storage medium on which a computer program is stored, which is used in a complex event detection device. When the computer program is executed by a processor, the complex event detection method described above is implemented.

Embodiments of the present application provide a complex event detection method, device, and storage medium. The complex event detection method includes: obtaining an operation event stream. The number of events in the operation event stream is multiple; binding each event in the operation event stream. Define multiple event rules and multiple event determination logic; use the calculation engine to determine the match between the operation event stream and the bound multiple event rules; in the case of matching, filter events from the operation event stream whose operation time is less than the target The target event of the water level; the target water level is a time determined based on the operation time of multiple events corresponding to the operation event stream; the determination result of the target event is determined based on the multiple event determination logic, and the target event is determined to be a complex event based on the determination result. Using the above method to implement the solution, the complex event detection device has multiple event rules and multiple event determination logics in the form of expressions. When adding complex event detection rules and event determination logic, the newly added complex event detection rules and event determination logic can be directly added. The complex event detection rules and event determination logic are broadcast to the calculation engine, so that the calculation engine can be used to parse and calculate the new rules in the form of expressions. There is no need to add code for the rule calculation unit, and there is no need to compile the code. Run, which improves the speed when adding complex event detection rules.

Description of the drawings

The accompanying drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application, and together with the description, are used to explain the technical solutions of the present application.

Figure 1 is a schematic diagram of a complex event detection method in the prior art provided by an embodiment of the present application;

Figure 2 is a flow chart of a complex event detection method provided by an embodiment of the present application;

Figure 3 is a flow chart of an exemplary complex event detection method provided by an embodiment of the present application;

Figure 4 is a schematic diagram of an exemplary complex event detection provided by an embodiment of the present application;

Figure 5 is a schematic structural diagram of a complex event detection device provided by an embodiment of the present application;

Figure 6 is a schematic diagram 2 of the composition and structure of a complex event detection device provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

The current mainstream Flink real-time complex event detection technology steps are shown in Figure 1. Flink reads complex event detection rules to form a rule stream, then converts the rules into a broadcast stream through broadcast operators, and then binds the broadcast stream to the event stream, and then The keyBy operator is used to group and calculate according to a fixed field (usually user ID, device ID). During the calculation process, all rules are first traversed, and then each rule is identified through the JAVA reflection mechanism, and then each event is sent to the corresponding pre- Calculate in the defined n rule calculation units to obtain the calculation results. Specifically, it includes the following steps:

1. Flink reads complex event detection rules to form a rule flow. The rule format is generally a string in JSON format. For example, the "dcp user privacy authorization" complex event detection rule is defined as:

{

"rule_id":"dcp_privacy_grant",

"rule_name":"dcp user privacy authorization",

"rule_begin_key":"defined_name",

"rule_begin_value":"dcp-login-succ",

"rule_middle_key":"action_name",

"rule_middle_value":"read-privacy-statement",

"rule_end_key":"click_name",

"rule_end_value":"agree",

"rule_calc_id":"calc_grant"

}

2. Flink extracts the event time field in the real-time event stream to generate the watermark of the real-time event stream.

3. Flink converts the rule stream into a broadcast stream through the broadcast operator.

4. Flink binds the rule stream to the event stream through the Connect operator to generate an event stream bound to the rules. For example, an event in the event stream is:

An event in the event stream bound to a rule is:

5. Flink uses the keyBy operator to extract a fixed field (usually user id, device id, etc.) to convert the event stream of bound rules into a group stream for group calculation, including the following specific steps:

5.1. Flink traverses each rule of each event in the packet stream.

5.2. Flink uses the JAVA reflection mechanism to identify complex event detection rules bound to each event, including rule id, rule name, start rule key, start rule value, intermediate rule key, intermediate rule value, end rule key, end rule value, rule Calculate unit id and other information.

5.3. Flink sends each event to the corresponding predefined n rule calculation units for calculation and obtains the calculation results. Among them, the calculation method of complex events is predefined in each rule calculation unit. For example, the complex event detection calculation unit calc_grant of "User Privacy Authorization" is defined as: login success event + read agreement action event + click agree button event. If these three events occur, YES will be output, indicating that "User Privacy Authorization" has been detected. Complex events.

Problems existing in the prior art can be specifically solved by the methods in the following embodiments.

The embodiment of the present application provides a complex event detection method. The complex event detection method is applied to a complex event detection device. Figure 2 is a flow chart of the complex event detection method provided by the embodiment of the present application. As shown in Figure 2, Complex event detection methods can include:

S101. Obtain the operation event stream. The number of events in the operation event stream is multiple.

The complex event detection method provided by the embodiment of the present application is suitable for the scenario of detecting complex events.

In the embodiment of the present application, the complex event detection device can be implemented in various forms. For example, the complex event detection device described in this application may include a device such as a server.

In the embodiment of this application, when the embedded system detects a real-time operation event, the embedded system reports the operation event to the Kafka message queue. The complex event detection device can regularly obtain multiple messages from the Kafka message queue. operation events to obtain the operation event stream.

It should be noted that the complex event detection device can also obtain multiple operation events from the Kafka message queue in a preset time period to obtain the operation event stream; the complex event detection device can also obtain operation events in the Kafka message queue. When the number is the preset number, the complex event detection device obtains multiple operation events from the Kafka message queue and obtains the operation event stream; the details can be determined according to the actual situation, and the embodiments of this application are not limited to this.

It should be noted that the preset time period can be a time period configured in the complex event detection device; the preset time period can also be a time period input by the user into the complex event detection device; a specific complex event detection device obtains the preset time The segment method can be determined according to the actual situation, and the embodiment of the present application does not limit this.

It should be noted that the preset number can be the number configured in the complex event detection device; the preset number can also be the number transmitted to the complex event detection device by other devices; the specific method for the complex event detection device to obtain the preset number can be The determination is made based on the actual situation, which is not limited in the embodiments of this application.

For example, taking the small loan product (DCP) as an example, the operation event detection process of detecting "DCP user authorization behavior is time-consuming": the hidden system reports the operation event to the Kafka message queue in real time, and Flink consumes the real-time information in Kafka in real time. The operation events form a real-time operation event stream of hidden events. An operation event in the operation event stream can be:

It should be noted that each operation event in the operation event stream carries the event operation time.

S102. Bind multiple event rules and multiple event determination logic to each event in the operation event stream.

In the embodiment of the present application, after the complex event detection device obtains the operation event stream, the complex event detection device can bind multiple event rules and multiple event determination logics to each event in the operation event stream.

In the embodiment of the present application, after the complex event detection device binds multiple event rules and multiple event determination logics to each event in the operation event stream, when multiple event rules or multiple event determination logics are changed, , the complex event detection device also changes multiple event rules or multiple event determination logics bound to the operation events at the same time.

It should be noted that multiple event rules may have a one-to-one correspondence with multiple event determination logics, and multiple event rules may not have a one-to-one correspondence with multiple event determination logics; the specific determination may be based on the actual situation. Examples of this application There is no limit to this.

In the embodiment of this application, the complex event detection device can use Flink to bind multiple event rules and multiple event determination logics to each event in the operation event stream through the Connect operator; the complex event detection device can also use other methods. Bind multiple event rules and multiple event determination logic to each event in the operation event stream; a specific complex event detection device binds multiple event rules and multiple event determination logic to each event in the operation event stream. The method can be determined according to actual conditions, and is not limited in the embodiments of this application.

In the embodiment of the present application, before the complex event detection device binds multiple event rules and multiple event determination logics to each event in the operation event stream, the complex event detection device will also receive multiple configured event rules and multiple event determination logics. Event determination logic; the complex event detection device broadcasts multiple event rules and multiple event determination logic to the computing engine.

In the embodiment of the present application, the multiple event rules and multiple event determination logics can be rules and logic defined using Aviator expressions; the multiple event rules and multiple event determination logics can also be rules and logic defined in other ways. Logic; the details can be determined according to the actual situation, and the embodiments of this application do not limit this.

In the embodiment of the present application, after the complex event detection device receives the configured multiple event rules and multiple event determination logics, the complex event detection device can store the multiple event rules and multiple event determination logics in the database.

It should be noted that multiple event rules and multiple event determination logic include: rule number, whether the rule is enabled, creation time, last update time, product number, indicator category, indicator name, field name where the indicator is stored in the database, complex Event categories, complex event subcategories, rule start conditions, rule intermediate conditions, rule end conditions, calculation logic and other information.

In the embodiment of this application, the complex event detection device can use Flink to read the Aviator expression stored in the database every n seconds as multiple event rules and multiple event determination logic.

In the embodiment of this application, taking DCP as an example, Aviator expressions are used to define multiple event rules and multiple event determination logics. Flink reads the Aviator expressions stored in the database every n seconds to form "multiple event rules and multiple event determination logics". Each event determines the logic":

In the embodiment of this application, the complex event detection device can use the Flink broadcast operator to broadcast multiple event rules and multiple event determination logics. The complex event detection device can also use other broadcast methods to broadcast multiple event rules and multiple event determination logics. The event determination logic is broadcast; the specific method of broadcasting multiple event rules and multiple event determination logics to the computing engine by the complex event detection device can be determined according to the actual situation, and is not limited in the embodiments of the present application.

In the embodiment of this application, taking DCP as an example, a broadcast operator is used to broadcast the "rules and calculation logic flow" to form a broadcast stream.

S103. Use the calculation engine to determine the matching between the operation event stream and the multiple bound event rules.

In the embodiment of the present application, after the complex event detection device binds multiple event rules and multiple event determination logics to each event in the operation event stream, the complex event detection device can use the computing engine to determine the operation event flow and binding Matching between multiple event rules.

In the embodiment of the present application, the calculation engine may be the Aviator expression engine; the calculation engine may also be other engines used for calculation functions; the details may be determined according to the actual situation, which is not limited in the embodiment of the present application.

In the embodiment of this application, the Aviator expression engine can be called in stateFunc, and the Aviator expression engine can be used to determine the match between the operation event stream and the multiple bound event rules.

In the embodiment of the present application, the process of the complex event detection device using the computing engine to determine the matching between the operation event stream and the multiple bound event rules includes: the complex event detection device obtains the preset fields in the operation event stream, Obtain multiple field information; group the operation event stream according to the multiple field information to obtain multiple groups of operation events; determine the matching between each group of operation events in the multiple groups of operation events and the multiple bound event rules, Until the matching between multiple sets of operation events and multiple bound event rules is determined.

In the embodiment of the present application, the preset field can be a field configured in the complex event detection device; the preset field can also be a field transmitted to the complex event detection device by other devices; the preset field can also be a field configured by the complex event detection device. Fields obtained in other ways; the specific way in which the complex event detection device obtains the preset fields can be determined according to the actual situation, and this is not limited in the embodiment of this application number.

It should be noted that the preset field can be user id, the preset field can also be device id, and the preset field can also be other field information; the specific preset field can be determined according to the actual situation. The embodiments of this application are This is not a limitation.

It should also be noted that each operation event in the operation event stream carries field information corresponding to the preset field.

In the embodiment of this application, the complex event detection device can use Flink's map operator to customize a state calculation function stateFunc, so that stateFunc can be used to traverse multiple event rules bound to itself for each group of operating events in multiple groups. Determine logic with multiple events. Taking DCP as an example, each operation event in multiple groups of operation events that are bound to multiple event rules and multiple event determination logics traverses the multiple event rules and multiple event determination logics bound to itself.

In the embodiment of the present application, the complex event detection device groups the operation event stream according to multiple field information, and the process of obtaining multiple groups of operation events includes: the complex event detection device separately verifies the multiple field information, and obtains multiple Check value; the complex event detection device performs hash processing on multiple check values to obtain multiple hash information; the complex event detection device groups the operation event stream according to the multiple hash information to obtain multiple groups of operation events.

In the embodiment of the present application, the complex event detection device performs hash processing on multiple check values to obtain multiple hash information, including the complex event detection device modulo operator parallelism based on multiple check values. Process to obtain multiple hash information; the complex event detection device can also obtain multiple hash information based on multiple check values in other ways; the specific determination can be based on the actual situation, and the embodiment of the present application is Not limited.

It should be noted that the operator parallelism can be the number of central processing units (Central Processing Unit, CPU) in the complex event detection device. For example, the operator parallelism may be a 200-core central processing unit.

In the embodiment of the present application, the complex event detection device separately verifies multiple field information to obtain multiple check values. The complex event detection device can use the CRC32 verification method to separately verify multiple field information. , to obtain multiple check values; the complex event detection device can also use other verification methods to separately verify multiple field information and obtain multiple check values; the specifics can be determined according to the actual situation, and this application implements This example does not limit this.

It should be noted that multiple field information corresponds to multiple check values one-to-one, that is, one field information corresponds to one check value.

In the embodiment of this application, the hash information is hashId. Multiple hash information corresponds to multiple check values one-to-one, that is, one hash information corresponds to one check value.

In the embodiment of this application, taking DCP as an example, extract the CRC32 check value of the user_id field and perform modulo operation on the parallelism degree of the operator 100, that is, hashId=mod(crc32('xxxxxxxxxxxxxxxxx001'),100)+1 to obtain the value hashId=91.

In the embodiment of this application, the complex event detection device can use multiple hash information as input parameters of Flink's keyBy operator to generate multiple sets of operation events.

In the embodiment of this application, taking DCP as an example, the keyBy operator is used to group the operation event streams bound to multiple event rules according to hashId and generate multiple groups of operation events grouped by key. For example, the user information is user_id=xxxxxxxxxxxxxxxxx00 operation The event and the operation event with user information user_id=simulator_dcp_delta4_sit631 have the same hashId, that is, the operation events of these two users will be grouped into the same group, so even if the user with user_id=xxxxxxxxxxxxxxxxx001 has no subsequent operation events, as long as the other hashId=91 If the user has subsequent operation events, the water level of the current group can be raised, thereby triggering the calculation of the operation event data of the user with user_id=xxxxxxxxxxxxxxxxx001.

S104. In the case of matching, filter target events whose event operation time is less than the target water level from the operation event stream; the target water level is a time determined based on multiple event operation times corresponding to the operation event stream.

In the embodiment of the present application, after the complex event detection device uses the calculation engine to determine the match between the operation event stream and the multiple bound event rules, the complex event detection device determines the match from the operation event stream. Filter the target events whose event operation time is less than the target water level.

In the embodiment of the present application, if there is a match, the complex event detection device writes the operation event into the status list; if there is no match, the complex event detection device deletes the operation event.

In the embodiment of the present application, the complex event detection device selects target events whose operation time is less than the target water level from the operation event stream, and can obtain the first operation event in the status list for the complex event detection device. The complex event detection device Target events whose event operation time is less than the target water level are filtered from the first operation events.

It should be noted that the first operation event is all operation events added to the status list.

In the embodiment of this application, taking DCP as an example, that is, determining whether the rule start condition, rule intermediate condition, and rule end condition are met. For the begin event and end event in the following "event flow of binding rules and calculation logic", Aviator The expression engine will determine that the results of RULE_BEGIN and RULE_END are true and write these two events into the state.

begin event:

end event:

In the embodiment of the present application, the complex event detection device filters the event operation time from the operation event stream before the target event that is smaller than the target water level. The complex event detection device also obtains multiple event operation times corresponding to the operation event stream, and based on the multiple event operation times, Each event operation time generates multiple water levels; the complex event detection device merges multiple water levels to obtain a merged water level; the complex event detection device selects the target with the smallest water level value from the merged water level and the preset water level Water level.

In the embodiment of this application, Flink extracts the event operation time field in the operation event stream to generate the corresponding water mark.

In the embodiment of the present application, the complex event detection device generates multiple watermarks based on multiple event operation times. The complex event detection device can generate multiple watermarks by using the preset duration of tolerating disorder as a watermark generation strategy. ; It is also possible for the complex event detection device to use multiple event operation times as multiple water levels; the specific way in which the complex event detection device generates multiple water levels according to the multiple event operation times can be determined according to the actual situation. Embodiments of the present application There is no limit to this.

In the embodiment of the present application, the preset duration can be the duration configured in the complex event detection device; the preset duration can also be the duration transmitted to the complex event detection device by other devices; the specific complex event detection device obtains the preset duration The method can be determined according to actual conditions, and is not limited in the embodiments of this application.

It should be noted that the preset duration can be 5 seconds, the preset duration can also be 3 seconds, and the preset duration can also be 1 second. The specific preset duration can be determined according to the actual situation, and this is not the case in the embodiments of this application. limited.

In the embodiment of this application, taking DCP as an example, Flink extracts the timestamp field in the real-time event stream as the input parameter of the watermark generator, and uses the tolerance of disorder for 5 seconds as the watermark generation strategy to generate the event pipeline watermark.

The water level line shape is as follows:

Watermark: 1646877595000

The corresponding event data format is as follows:

In the embodiment of the present application, the preset water level can be the water level configured in the complex event detection device; the preset water level can also be the water level transmitted to the complex event detection device by other devices; the specific complex event detection device is preset The method of setting the water level can be determined according to actual conditions, and is not limited in the embodiments of the present application.

It should be noted that the preset water level can be the maximum value of the Long type, and the preset water level can also be other values; the specific preset water level can be determined according to the actual situation, and this is not limited in the embodiments of the present application.

In the embodiment of this application, taking DCP as an example, 2 raised to the power of 63 -1 is used as the input parameter of the water level generator to generate a preset water level. The preset water level can be: watermark:0x7fffffffffffffffL.

In the embodiment of this application, taking DCP as an example, the operation event stream and multiple event rules and multiple event determination logics are connected through the connect operator to generate an operation event stream that binds multiple event rules and multiple event determination logics. Extract the user id field in the operation event stream and calculate the CRC32 check value, then perform a modular operation on the operator parallelism to generate a hashId, and take the water level of the event stream (piece operation time) and the water level of the rule flow (preset water level) The smaller value of the two is used as the target water level.

In the embodiment of the present application, the complex event detection device merges multiple water levels to obtain the merged water level, which includes: the complex event detection device obtains the first water level with the largest water level value from the multiple water levels; The complex event detection device uses the first water level as the merged water level.

S105. Determine the determination result of the target event according to multiple event determination logics, so as to determine the target event as a complex event according to the determination result.

In the embodiment of the present application, after the complex event detection device filters target events whose event operation time is less than the target water level from the operation event stream, the complex event detection device can determine the determination result of the target event based on multiple event determination logics. The determination result determines that the target event is a complex event.

In the embodiment of the present application, the complex event detection device determines the determination result of the target event based on multiple event determination logics. The complex event detection device may call the Aviator expression engine to determine the determination result of the target event based on multiple event determination logics.

In the embodiment of this application, the complex event detection device reads all the status data of the current hashId (corresponding to the same hashId) in stateFunc. The operation event corresponding to the status data is the first operation event. The complex event detection device starts from the first operation event. The events are screened for target events whose event operation time is less than the target water level, and complex event detection is performed on the target event, and the Aviator expression engine is called to parse and execute the calculation logic to generate the final result of the target event.

In the embodiment of this application, taking DCP as an example, that is, reading all the status data of the current group with hashId = 91, and then filtering the target events whose event operation time is less than the target water level in the first operation event corresponding to the status data. Events with the same user_id and the same RULE_ID are then put together for calculation. The event rules are determined according to the complex event category, complex event subcategory, and event determination logic, such as CEP_TYPE=deltaTime and CEP_SUBTYPE=1 and CEP_CALC_EXPRESS=END.timestamp- BEGIN.timestamp, represents the time difference between two non-strictly continuous events. The event determination logic after complex event detection = the event time of the end event - the event time of the start event. Call the Aviator expression engine to parse and execute the event determination logic, that is, calc_grant_delta_time =1646877700000-1646877600000=1000000 milliseconds.

In the embodiment of the present application, the complex event detection device clears the status of the operation events corresponding to the calculated complex events in the stateFunc, as well as the status of the operation events that do not match the event rules. Taking DCP as an example, that is, the status of the operation events corresponding to the calculated complex events is deleted, as well as the status of the operation events that do not match the event rules.

In this embodiment of the present application, an exemplary complex event detection method is shown in Figure 3:

S30. The complex event detection device receives the configured multiple event rules and multiple event determination logics, and broadcasts the multiple event rules and multiple event determination logics to the computing engine.

S31. The complex event detection device obtains multiple event operation times corresponding to the operation event stream, and generates multiple watermarks based on the multiple event operation times.

S32. The complex event detection device merges multiple water levels to obtain a merged water level.

S33. The complex event detection device selects the target water level with the smallest water level value from the combined water level and the preset water level.

S34. The complex event detection device obtains the operation event stream, and binds multiple event rules and multiple event determination logics to each event in the operation event stream.

It should be noted that the number of events in the operation event stream is multiple.

S35. The complex event detection device obtains preset fields in the operation event stream and obtains multiple field information; and performs verification on the multiple field information respectively to obtain multiple verification values.

S36. The complex event detection device performs hash processing on multiple check values to obtain multiple hash information; and groups the operation event streams according to the multiple hash information to obtain multiple groups of operation events.

S37. The complex event detection device determines the matching between each group of operation events in the multiple groups of operation events and the multiple bound event rules until the matching between the multiple groups of operation events and the multiple bound event rules is determined. Matchability.

S38. In the case of matching, the complex event detection device filters target events whose event operation time is less than the target water level from the operation event stream.

It should be noted that the target water level is a time determined based on multiple event operation times corresponding to the operation event stream.

S39. The complex event detection device determines the determination result of the target event based on multiple event determination logics, and determines the target event as a complex event based on the determination result.

Illustratively, as shown in Figure 4: Taking DCP as an example, the complex event detection device detects the operation event detection process of "DCP user authorization behavior is time-consuming": the hidden system reports the operation event to the kafka message queue in real time, and Flink consumes it in real time The real-time operation events in Kafka form a real-time operation event stream of hidden events. Flink extracts the timestamp field in the real-time event stream as the input parameter of the watermark generator, and uses the tolerance of disorder for 5 seconds as the watermark generation strategy to generate the bitlines corresponding to the event pipeline. Aviator expressions are used to define multiple event rules and multiple event determination logic. Flink reads the Aviator expression stored in the database every n seconds to form "multiple event rules and multiple event determination logic". Use 2 to the power of 63 -1 as the input parameter of the water level generator to generate a preset water level. The broadcast operator is used to broadcast "multiple event rules and multiple event determination logic (rules and calculation logic flow)" to form a broadcast stream. Extract the CRC32 check value of the user_id field and perform modulo operation on the operator's parallelism degree of 100 to obtain hashId=91. The connect operator connects (binds) the operation event stream and multiple event rules with multiple event determination logic to generate an operation event stream that binds multiple event rules and multiple event determination logic (binds the operation event stream with the rules and Calculate the logical flow), extract the user id field in the operation event stream and calculate the CRC32 check value, then perform a modular operation on the operator parallelism to generate a hashId, and take the watermark of the event stream (piece operation time) and the watermark of the rule stream ( The smaller of the two values (default water level) is used as the target water level. The keyBy operator is used to group the operation event streams bound to multiple event rules according to hashId and generate multiple groups of operation events grouped by key. For example, the operation event with user information = user_id = xxxxxxxxxxxxxxxxx00 and the operation event with user information = user_id = simulator_dcp_delta4_sit631 are With the same hashId, that is, the operation events of these two users will be grouped into the same group. In this way, even if the user with user_id=xxxxxxxxxxxxxxxxx001 has no subsequent operation events, as long as other users with hashId=91 have subsequent operation events, the current group can be triggered. The water level is raised, thereby triggering the calculation of the operation event data of the user with user_id=xxxxxxxxxxxxxxxxx001. For each operation event in multiple groups of operation events bound to multiple event rules and multiple event determination logics, each operation event traverses the multiple event rules and multiple event determination logics bound to itself. Taking DCP as an example, it determines whether the rule start condition, rule intermediate condition, and rule end condition are met. For the begin event and end event in the following "Event Flow of Binding Rules and Calculation Logic", the Aviator expression engine will determine whether RULE_BEGIN and RULE_END evaluates to true and writes both events to the state. Read all the status data of the current group with hashId = 91, then filter the target events whose event operation time is less than the target water level in the first operation event corresponding to the status data, and then put the events with the same user_id and the same RULE_ID together. Calculation is performed, and the event rules are determined according to the complex event category, complex event subcategory, and event determination logic. For example, CEP_TYPE=deltaTime and CEP_SUBTYPE=1 and CEP_CALC_EXPRESS=END.timestamp-BEGIN.timestamp, indicating two non-strictly consecutive events. Time difference, event determination logic after complex event detection = event time of the end event - event time of the start event, call the Aviator expression engine to parse and execute the event determination logic (get the determination result of the target event). Delete the status of the operation events corresponding to the calculated complex events, as well as the status of the operation events that do not match the event rules.

It can be understood that the complex event detection device has multiple event rules and multiple event determination logics in the form of expressions. When complex event detection rules and event determination logic are added, the newly added event detection rules and event determination logic can be directly added. Complex event detection rules and event determination logic are broadcast to the calculation engine, so that the calculation engine can be used to parse and calculate new rules in the form of expressions. There is no need to add code for the rule calculation unit, nor to compile and run the code. , improves the speed when adding complex event detection rules.

Based on the same inventive concept as the above complex event detection method, the embodiment of the present application provides a complex event detection device 1, corresponding to a complex event detection method; Figure 5 is a complex event detection device provided by the embodiment of the present application. Schematic diagram 1 of the composition structure of the complex event detection device 1 may include:

The acquisition part 11 is configured to acquire an operation event stream, where the number of events in the operation event stream is multiple;

The binding part 12 is configured to bind multiple event rules and multiple event determination logic to each event in the operation event stream;

The determination part 13 is configured to use the calculation engine to determine the matching between the operation event stream and the multiple bound event rules; determine the determination result of the target event according to the multiple event determination logic, so as to determine the result of the target event according to the determination result. Determine that the target event is a complex event;

The filtering part 14 is configured to filter the target events whose event operation time is less than the target water level from the operation event stream in the case of matching; the target water level is a plurality of events corresponding to the operation event stream. The operating time is determined by the time.

In some embodiments of the present application, the device further includes a receiving part and a broadcasting part;

The receiving part is configured to receive multiple configured event rules and multiple event determination logics;

The broadcast part is configured to broadcast the plurality of event rules and the plurality of event determination logic to the computing engine.

In some embodiments of the present application, the device further includes a grouping part;

The acquisition part 11 is configured to acquire preset fields in the operation event stream and obtain multiple field information;

The grouping part is configured to group the operation event stream according to the plurality of field information to obtain multiple groups of operation events;

The determination part 13 is configured to determine the matching between each group of operation events in the plurality of groups of operation events and a plurality of bound event rules, until it is determined that the plurality of groups of operation events and the plurality of bound event rules are matched. Matching between event rules.

In some embodiments of the present application, the device further includes a verification part and a processing part;

The verification part is configured to verify the plurality of field information respectively and obtain multiple verification values;

The processing part is configured to perform hash processing on the multiple check values to obtain multiple hash information;

The grouping part is configured to group the operation event stream according to the plurality of hash information to obtain the plurality of groups of operation events.

In some embodiments of the present application, the acquisition part 11 is configured to acquire multiple event operation times corresponding to the operation event stream, and generate multiple watermarks based on the multiple event operation times;

The processing part is configured to merge the plurality of water levels to obtain a merged water level;

The screening part 14 is configured to screen the target water level with the smallest water level value from the merged water level and the preset water level.

In some embodiments of the present application, the acquisition part 11 is configured to acquire the first water level with the largest water level value from the multiple water levels; and use the first water level as the merged water level.

It should be noted that in practical applications, the above-mentioned acquisition part 11, binding part 12, determination part 13 and filtering part 14 can be implemented by the processor 15 on the complex event detection device 1, specifically a CPU (Central Processing Unit, central processing unit). processor), MPU (Microprocessor Unit, microprocessor), DSP (Digital Signal Processing, digital signal processor) or field programmable gate array (FPGA, Field Programmable Gate Array), etc.; the above data storage can be implemented by a complex event detection device 1 implemented on the memory 16.

The embodiment of the present application also provides a complex event detection device 1. As shown in Figure 6, the complex event detection device 1 includes: a processor 15, a memory 16 and a communication bus 17. The memory 16 passes through the communication bus. 17 communicates with the processor 15, and the memory 16 stores a program executable by the processor 15. When the program is executed, the complex event detection method as described above is executed by the processor 15.

In practical applications, the above-mentioned memory 16 can be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), flash memory (flash memory), hard disk (Hard Disk Drive, HDD) or solid-state drive (Solid-State Drive, SSD); or a combination of the above types of memory, and to the processor 15 Provide instructions and data.

The embodiment of the present application provides a computer-readable storage medium with a computer program on it. When the program is executed by the processor 15, the complex event detection method as described above is implemented.

It can be understood that the complex event detection device has multiple event rules and multiple event determination logics in the form of expressions. When complex event detection rules and event determination logic are added, the newly added event detection rules and event determination logic can be directly added. Complex event detection rules and event determination logic are broadcast to the calculation engine, so that the calculation engine can be used to parse and calculate the new rules in the form of expressions. There is no need to add code for the rule calculation unit, and there is no need to compile and run the code. , improves the speed when adding complex event detection rules.

Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk storage and optical storage, etc.) embodying computer-usable program code therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

The above descriptions are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application.

Industrial applicability

Claims

A complex event detection method, the method includes:

Obtain an operation event stream, where the number of events in the operation event stream is multiple;

Bind multiple event rules and multiple event determination logic to each event in the operation event stream;

Using a calculation engine to determine the match between the operation event stream and the multiple bound event rules;

In the case of matching, target events whose event operation time is less than a target water level are screened from the operation event stream; the target water level is a time determined based on multiple event operation times corresponding to the operation event stream;

The determination result of the target event is determined according to the plurality of event determination logics, so that the target event is determined to be a complex event according to the determination result.
The method according to claim 1, wherein before binding multiple event rules and multiple event determination logics to each event in the operation event stream, the method further includes:

Receive configured multiple event rules and multiple event determination logic;

The plurality of event rules and the plurality of event determination logic are broadcast to the computing engine.
The method according to claim 1, wherein the utilizing a computing engine to determine the matching between the operation event stream and a plurality of bound event rules includes:

Obtain preset fields in the operation event stream and obtain multiple field information;

Group the operation event stream according to the plurality of field information to obtain multiple groups of operation events;

Determine the matching between each group of operation events in the plurality of groups of operation events and the plurality of bound event rules until the matching between the plurality of groups of operation events and the plurality of bound event rules is determined. .
The method according to claim 3, wherein the operation event stream is grouped according to the plurality of field information to obtain multiple groups of operation events, including:

Perform verification on the multiple field information respectively to obtain multiple verification values;

Perform hash processing on the multiple check values to obtain multiple hash information;

The operation event stream is grouped according to the plurality of hash information to obtain the plurality of groups of operation events.
The method according to claim 1, wherein filtering the event operation time from the operation event stream before the target event that is less than the target water level, the method further includes:

Obtain multiple event operation times corresponding to the operation event stream, and generate multiple water levels based on the multiple event operation times;

Merge the plurality of water levels to obtain a merged water level;

From the combined water level and the preset water level, a target water level with the smallest water level value is selected.
The method according to claim 5, wherein said merging the plurality of water levels to obtain a merged water level includes:

Obtain the first water level with the largest water level value from the plurality of water levels;

The first water level is regarded as the merged water level.
A complex event detection device, the device includes:

The acquisition part is configured to acquire an operation event stream, where the number of events in the operation event stream is multiple;

The binding part is configured to bind multiple event rules and multiple event determination logic to each event in the operation event stream;

The determination part is configured to use the calculation engine to determine the matching between the operation event stream and the multiple bound event rules; determine the determination result of the target event according to the multiple event determination logic, so as to determine the determination result according to the determination result. The target event is a complex event;

The filtering part is configured to, in the case of matching, filter the target events whose event operation time is less than the target water level from the operation event stream; the target water level is a plurality of event operations corresponding to the operation event stream. Time determined time.
The device according to claim 7, wherein the device further includes a receiving part and a broadcasting part;

The receiving part is configured to receive multiple configured event rules and multiple event determination logics;

The broadcast part is configured to broadcast the plurality of event rules and the plurality of event determination logic to the computing engine.
A complex event detection device, the device includes:

A memory, a processor and a communication bus. The memory communicates with the processor through the communication bus. The memory stores a complex event detection program executable by the processor. When the complex event detection program is When executed, the method according to any one of claims 1 to 7 is executed by the processor.
A storage medium with a computer program stored thereon, which is used in a complex event detection device. When the computer program is executed by a processor, the method of any one of claims 1 to 7 is implemented.