CN112506939A - CEP rule updating method based on Flink stream engine - Google Patents
CEP rule updating method based on Flink stream engine Download PDFInfo
- Publication number
- CN112506939A CN112506939A CN202011443193.XA CN202011443193A CN112506939A CN 112506939 A CN112506939 A CN 112506939A CN 202011443193 A CN202011443193 A CN 202011443193A CN 112506939 A CN112506939 A CN 112506939A
- Authority
- CN
- China
- Prior art keywords
- rule
- event
- updating
- operator
- stream
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012098 association analyses Methods 0.000 claims description 7
- 238000010219 correlation analysis Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a CEP rule updating method based on a Flink stream engine, relating to the technical field of CEP rule updating methods and comprising the following steps: each task loads an initial rule from an external file, stores the initial rule in a memory according to a key value pair, accesses an event stream from an external message engine, and analyzes the event stream according to the loaded rule; the rule data needing to be updated is obtained from an external cache or a system, then broadcast and combined with the event stream. The task of the invention accesses the event stream from the external message engine, analyzes according to the loaded rule, acquires the rule data to be updated from the external cache or the system, then broadcasts, only needs to go to the rule base to acquire the rule once, and simultaneously converts the updating rule into the event stream with a special mark, if the parallelism of the downstream operator is only one, an event is generated, if a plurality of events exist, a plurality of corresponding events are generated, the type field of the event is changed into the rule type, the number of the event is changed into the number of the rule, and other corresponding fields are also updated.
Description
Technical Field
The invention relates to a CEP rule updating method, relates to the technical field of CEP rule updating methods, and particularly relates to a CEP rule updating method based on a Flink stream engine.
Background
The data stream engine updating rule is generally read from a cache, generates a stream from the rule, then combines the stream with an event stream, replaces the original rule on a combination node, and then can analyze the stream by using a new rule. The following problems exist in the prior art:
1. the existing rule updating method can only update the operator nodes of the combined stream, but cannot update the operator nodes at the downstream of the combined stream;
2. the conventional rule updating method has no problem in the general rule updating, but needs to rewrite the service code for the CEP rule.
Disclosure of Invention
The invention provides a CEP rule updating method based on a Flink stream engine, which aims to realize the capability of updating downstream operator nodes and solve the problem that only operator nodes combined by stream can be updated; the other purpose is to solve the problem that the CEP rule needs to be updated after the service code is rewritten, so as to simplify the updating process.
In order to solve the technical problems, the technical scheme adopted by the invention is as follows:
one aspect of the method is a CEP rule updating method based on a Flink streaming engine, and the method comprises the following steps:
the method comprises the following steps: each task loads an initial rule from an external file, stores the initial rule in a memory according to a key value pair, accesses an event stream from an external message engine, and analyzes the event stream according to the loaded rule;
step two: acquiring rule data needing to be updated from an external cache or a system, then broadcasting, and combining with the event stream, wherein each task acquires a combined stream at the operator node;
step three: according to the number of the updating rule, comparing with the key of the operator node in the second step, if the same, converting the updating rule into an event stream with a special mark, if a plurality of concurrent nodes exist at the moment, copying a plurality of rule events according to the rule, setting different values of specified fields, and then delivering the event stream to a downstream operator of the task;
step four: and comparing the analyzed operator node with the existing rule of the node according to the number information and the version information of the special event stream, updating the rule if the rule is consistent with the existing rule, and performing real-time association analysis by using the new rule.
The technical scheme of the invention is further improved as follows: the initial rule in the first step is a plurality of rule lists containing rule numbers.
The technical scheme of the invention is further improved as follows: and comparing the number of the rule in the rule flow in the third step with the key of the rule of the operator node needing to be updated, and if the comparison result is inconsistent, ignoring the update rule.
The technical scheme of the invention is further improved as follows: and comparing the serial number of the updating rule in the rule flow in the third step with the key of the rule of the operator node needing to be updated, if the comparison result is consistent, converting the updating rule into an event flow with a special mark, correspondingly generating a plurality of events according to the parallelism of downstream operators, changing the type field of the event into the rule type, changing the serial number of the event into the serial number of the rule, and updating other corresponding fields.
The technical scheme of the invention is further improved as follows: and after the event stream is the event stream containing the special mark when the event stream is delivered to the downstream operator of the task in the third step, the downstream operator in the fourth step is compared with the rule used in the operator according to the event type and the serial number, if the type is the rule type, the operator updates the existing rule, and if the type is the event type, the event is analyzed by using the current rule.
Another aspect of the invention is a method for updating CEP rules based on a Flink streaming engine, which further includes the following steps:
s1: updating rules for the rule flow at a combined flow node combining the event flow and the rule flow, and analyzing the event flow after updating the rules;
s2: and after the updated rule analyzes the event stream, additionally compiling a processing service, and performing association analysis on each event by using the newly compiled processing service so as to complete the rule updating of all operator nodes.
Due to the adoption of the technical scheme, compared with the prior art, the invention has the technical progress that:
1. the invention provides a CEP rule updating method based on Flink flow engine, wherein the task accesses the event flow from the external message engine, analyzes according to the loaded rule, acquires the rule data to be updated from the external cache or system, then broadcasts, only needs to go to the rule base to acquire the rule once, at the same time, converts the updating rule into the event flow with special mark, generates a plurality of corresponding events according to the parallelism of the downstream operator, the type field of the event is changed into the rule type, the serial number of the event is changed into the serial number of the rule, other corresponding fields are updated, then delivers the event flow with special mark to the downstream operator of the task, the downstream operator compares with the rule used in the operator according to the event type and serial number, if the type is the rule type, the operator updates the existing rule, if the event type is the event type, the event is analyzed by the current rule, therefore, the updating of the downstream operator nodes is completed, and the purpose that the downstream operator nodes can be updated only by obtaining the rule once by the rule base in the rule updating is achieved.
2. The invention provides a CEP rule updating method based on a Flink streaming engine, the invention compares the number of the update rule with the key of the operator node in the second step, if the comparison result is the same, the update rule is converted into the event stream with the special mark, if a plurality of concurrent nodes exist at the moment, a plurality of rule events need to be copied according to the rules, different values of the designated fields are set, then delivering event stream to downstream operator of the task, comparing the analyzed operator node with the existing rule of the node according to the number information and version information of the special event stream, only if the conditions are consistent, the rule is updated, the new rule is used for carrying out real-time association analysis, the event stream delivered to the downstream operator can be directly subjected to association analysis in the whole process, extra business processing is not needed, the cost can be reduced, and meanwhile the working efficiency is improved.
3. The invention provides a CEP rule updating method based on a Flink stream engine, and the method does not need to pause and restart after the current task completes rule updating, the next task can directly load an initial rule and access an event stream from an external message engine, then the rule updating of a new task is carried out, and the restarting is not needed in the midway, thereby greatly saving time and functional working procedures and improving the working efficiency.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a block diagram of a rule updating process according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to the following examples:
example 1
As shown in fig. 1, the present invention provides a CEP rule updating method based on a Flink streaming engine, including the following steps:
the method comprises the following steps: each task loads an initial rule from an external file, the initial rule is a plurality of rule lists containing rule numbers, the rule lists are stored in a memory according to key value pairs, and in addition, the tasks access event streams from an external message engine and are analyzed according to the loaded rule;
step two: acquiring rule data needing to be updated from an external cache or a system, then broadcasting, and combining with the event stream, wherein each task acquires a combined stream at the operator node;
step three: comparing the number of the updating rule with the key of the operator node in the second step, if the same, converting the updating rule into an event stream with a special mark, if a plurality of concurrent nodes exist at the moment, copying a plurality of rule events according to the rule, setting different values of designated fields, then delivering the event stream to a downstream operator of the task, comparing the number of the updating rule in the rule stream with the key of the rule of the operator node to be updated, if the comparison result is inconsistent, ignoring the updating rule, on the other hand, comparing the number of the updating rule in the rule stream with the key of the rule of the operator node to be updated, if the comparison result is consistent, converting the updating rule into the event stream with the special mark, correspondingly generating a plurality of events according to the parallelism of the downstream operator, changing the type field of the event into the type of the rule, changing the number of the event into the number of the rule, other corresponding fields are updated, the event stream delivered to the downstream operator can be directly subjected to correlation analysis in the whole process, additional business processing is not needed, the cost can be reduced, and meanwhile the working efficiency is improved;
step four: the analyzed operator node is compared with the existing rule of the node according to the number information and version information of the special event stream, if the event stream is consistent with the existing rule of the node, the rule is updated, the new rule is used for real-time correlation analysis, after the event stream is delivered to the downstream operator of the task in the step three and is the event stream containing the special mark, the downstream operator in the step four is compared with the rule used in the operator according to the event type and the number, if the event type is the rule type, the existing rule is updated by the operator, if the event type is the event type, the event is analyzed by the current rule, so that the updating of the downstream operator node is completed, and the purpose that the downstream operator node can be updated only by obtaining the rule once by the rule base in the rule updating is achieved.
In this embodiment, after the current task completes the rule update, the restart is not required to be suspended, the next task can directly load the initial rule and access the event stream from the external message engine, and then the rule update of the new task is performed without the restart in the midway, so that the time and the functional workflow are greatly saved, and the working efficiency is improved.
Example 2
As shown in fig. 1, the present invention provides a method for updating CEP rules based on a Flink streaming engine, where the method for updating CEP rules based on a Flink streaming engine further includes the following steps:
s1: updating rules for the rule flow at a combined flow node combining the event flow and the rule flow, and analyzing the event flow after updating the rules;
s2: and after the updated rule analyzes the event stream, additionally compiling a processing service, and performing association analysis on each event by using the newly compiled processing service so as to complete the rule updating of all operator nodes.
In this embodiment, a task accesses an event stream from an external message engine, analyzes the event stream according to a loaded rule, acquires rule data to be updated from an external cache or a system, broadcasts the rule data, acquires a rule only by going to a rule base, converts the update rule into an event stream with a special sign, correspondingly generates a plurality of events according to the parallelism of a downstream operator, changes a type field of the event into a rule type, changes the serial number of the event into the serial number of the rule, updates other corresponding fields, delivers the event stream with the special sign to the downstream operator of the task, compares the downstream operator with the rule used in the operator according to the event type and the serial number, updates the existing rule if the type is the rule type, analyzes the event by using the current rule if the type is the event type, thereby completing the update of a downstream operator node, and only needs to go to the rule base to acquire the rule once for rule update, the purpose of the downstream operator node can be updated.
The present invention has been described in general terms in the foregoing, but it will be apparent to those skilled in the art that modifications and improvements can be made thereto based on the present invention. Therefore, modifications or improvements are within the scope of the invention without departing from the spirit of the inventive concept.
Claims (6)
1. A CEP rule updating method based on a Flink streaming engine is characterized in that: the method comprises the following steps:
the method comprises the following steps: each task loads an initial rule from an external file, stores the initial rule in a memory according to a key value pair, accesses an event stream from an external message engine, and analyzes the event stream according to the loaded rule;
step two: acquiring rule data needing to be updated from an external cache or a system, then broadcasting, and combining with the event stream, wherein each task acquires a combined stream at the operator node;
step three: according to the number of the updating rule, comparing with the key of the operator node in the second step, if the same, converting the updating rule into an event stream with a special mark, if a plurality of concurrent nodes exist at the moment, copying a plurality of rule events according to the rule, setting different values of specified fields, and then delivering the event stream to a downstream operator of the task;
step four: and comparing the analyzed operator node with the existing rule of the node according to the number information and the version information of the special event stream, updating the rule if the rule is consistent with the existing rule, and performing real-time association analysis by using the new rule.
2. The method for updating CEP rules based on the Flink streaming engine as claimed in claim 1, wherein: the initial rule in the first step is a plurality of rule lists containing rule numbers.
3. The method for updating CEP rules based on the Flink streaming engine as claimed in claim 1, wherein: and comparing the number of the rule in the rule flow in the third step with the key of the rule of the operator node needing to be updated, and if the comparison result is inconsistent, ignoring the update rule.
4. The method for updating CEP rules based on the Flink streaming engine as claimed in claim 1, wherein: and comparing the number of the update rule in the rule flow in the third step with the key of the rule of the operator node to be updated, converting the update rule into an event flow with a special mark if the comparison result is consistent, generating a plurality of corresponding events according to the parallelism of the downstream operators, and if the type field of an event is changed into the rule type, changing the number of the event into the number of the rule, and updating other corresponding fields.
5. The method for updating CEP rules based on the Flink streaming engine as claimed in claim 1, wherein: and after the event stream is the event stream containing the special mark when the event stream is delivered to the downstream operator of the task in the third step, the downstream operator in the fourth step is compared with the rule used in the operator according to the event type and the serial number, if the type is the rule type, the operator updates the existing rule, and if the type is the event type, the event is analyzed by using the current rule.
6. A CEP rule updating method based on a Flink streaming engine is characterized in that: the CEP rule updating method based on the Flink streaming engine further comprises the following steps:
s1: updating rules for the rule flow at a combined flow node combining the event flow and the rule flow, and analyzing the event flow after updating the rules;
s2: and after the updated rule analyzes the event stream, additionally compiling a processing service, and performing association analysis on each event by using the newly compiled processing service so as to complete the rule updating of all operator nodes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011443193.XA CN112506939A (en) | 2020-12-08 | 2020-12-08 | CEP rule updating method based on Flink stream engine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011443193.XA CN112506939A (en) | 2020-12-08 | 2020-12-08 | CEP rule updating method based on Flink stream engine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112506939A true CN112506939A (en) | 2021-03-16 |
Family
ID=74970989
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011443193.XA Pending CN112506939A (en) | 2020-12-08 | 2020-12-08 | CEP rule updating method based on Flink stream engine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112506939A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254106A (en) * | 2021-06-23 | 2021-08-13 | 中国平安人寿保险股份有限公司 | Task execution method and device based on Flink, computer equipment and storage medium |
| CN114116266A (en) * | 2022-01-27 | 2022-03-01 | 北京华品博睿网络技术有限公司 | Method and system for automatically splitting message based on stream calculation |
| CN117472517A (en) * | 2023-12-28 | 2024-01-30 | 广州睿帆科技有限公司 | Method for distributed processing of FTP files based on Flink |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100223499A1 (en) * | 2009-02-27 | 2010-09-02 | Microsoft Corporation | Fingerprinting event logs for system management troubleshooting |
| US20140108318A1 (en) * | 2012-10-11 | 2014-04-17 | Sap Ag | Synchronization for Context-Aware Complex Event Processing |
| US20160048565A1 (en) * | 2014-08-13 | 2016-02-18 | Software Ag | Systems and/or methods for investigating event streams in complex event processing (cep) applications |
| US9910874B1 (en) * | 2013-06-28 | 2018-03-06 | Emc Corporation | Scalable alerter for security information and event management |
| US20180075107A1 (en) * | 2016-09-15 | 2018-03-15 | Oracle International Corporation | Data serialization in a distributed event processing system |
| CN111309752A (en) * | 2019-11-26 | 2020-06-19 | 上海金融期货信息技术有限公司 | Service flow data processing oriented domain specific language assembly and operation method thereof |
| CN111444291A (en) * | 2020-03-27 | 2020-07-24 | 上海爱数信息技术股份有限公司 | Real-time data alarm method based on stream processing engine and rule engine |
| CN112000350A (en) * | 2020-07-29 | 2020-11-27 | 新华三大数据技术有限公司 | Dynamic rule updating method, device and storage medium |
-
2020
- 2020-12-08 CN CN202011443193.XA patent/CN112506939A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100223499A1 (en) * | 2009-02-27 | 2010-09-02 | Microsoft Corporation | Fingerprinting event logs for system management troubleshooting |
| US20140108318A1 (en) * | 2012-10-11 | 2014-04-17 | Sap Ag | Synchronization for Context-Aware Complex Event Processing |
| US9910874B1 (en) * | 2013-06-28 | 2018-03-06 | Emc Corporation | Scalable alerter for security information and event management |
| US20160048565A1 (en) * | 2014-08-13 | 2016-02-18 | Software Ag | Systems and/or methods for investigating event streams in complex event processing (cep) applications |
| US20180075107A1 (en) * | 2016-09-15 | 2018-03-15 | Oracle International Corporation | Data serialization in a distributed event processing system |
| CN111309752A (en) * | 2019-11-26 | 2020-06-19 | 上海金融期货信息技术有限公司 | Service flow data processing oriented domain specific language assembly and operation method thereof |
| CN111444291A (en) * | 2020-03-27 | 2020-07-24 | 上海爱数信息技术股份有限公司 | Real-time data alarm method based on stream processing engine and rule engine |
| CN112000350A (en) * | 2020-07-29 | 2020-11-27 | 新华三大数据技术有限公司 | Dynamic rule updating method, device and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 技术专家: "Flink任务实时获取并更新规则", pages 1 - 11, Retrieved from the Internet <URL:https://z.itpub.net/article/detail/1360AFF7862A16BD7189D858A45DE355> * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254106A (en) * | 2021-06-23 | 2021-08-13 | 中国平安人寿保险股份有限公司 | Task execution method and device based on Flink, computer equipment and storage medium |
| CN113254106B (en) * | 2021-06-23 | 2022-04-19 | 中国平安人寿保险股份有限公司 | Task execution method and device based on Flink, computer equipment and storage medium |
| CN114116266A (en) * | 2022-01-27 | 2022-03-01 | 北京华品博睿网络技术有限公司 | Method and system for automatically splitting message based on stream calculation |
| CN114116266B (en) * | 2022-01-27 | 2022-05-17 | 北京华品博睿网络技术有限公司 | Method and system for automatically splitting message based on stream calculation |
| CN117472517A (en) * | 2023-12-28 | 2024-01-30 | 广州睿帆科技有限公司 | Method for distributed processing of FTP files based on Flink |
| CN117472517B (en) * | 2023-12-28 | 2024-03-08 | 广州睿帆科技有限公司 | Method for distributed processing of FTP files based on Flink |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112506939A (en) | CEP rule updating method based on Flink stream engine | |
| CN112650766B (en) | Database data operation method, system and server | |
| US20040049768A1 (en) | Method and program for compiling processing, and computer-readable medium recoding the program thereof | |
| JP2006085740A (en) | Application software configuration method | |
| CN106708891A (en) | Network management data synchronizing method and device | |
| CN106557307B (en) | Service data processing method and system | |
| JP6784612B2 (en) | Analytical software management system and analytical software management method | |
| CN101980546B (en) | Intelligent network platform, service execution method and method for analyzing service abnormality | |
| US8615729B2 (en) | Extending existing model-to-model transformations | |
| CN110990055B (en) | Pull Request function classification method based on program analysis | |
| JP2008225898A (en) | Conversion device, conversion program, and conversion method | |
| CN104794179A (en) | Video quick indexing method and device based on knowledge tree | |
| CN113495723B (en) | Method, device and storage medium for calling functional component | |
| CN116048609A (en) | Configuration file updating method, device, computer equipment and storage medium | |
| CN115905353A (en) | Associated data export and import method, device, equipment and storage medium | |
| CN114895942A (en) | Application skin changing method, device, equipment and storage medium | |
| CN109857380B (en) | Workflow file compiling method and device | |
| CN113867714A (en) | Automatic code generation method adaptive to multiple languages | |
| CN111782641A (en) | Data error repairing method and system | |
| Jie et al. | A Model Driven method to represent Free Choice Petri Nets as Sequence Diagram | |
| CN111459583A (en) | Menu creating method and device | |
| JP3999949B2 (en) | Alias name search apparatus, alias name search method, and alias name search program | |
| JP2002116911A (en) | Automatic generation device for object-oriented program | |
| CN110618886A (en) | Version rollback method, mobile terminal and computer storage medium | |
| CN112306499B (en) | Program element extraction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |