WO2013097073A1

WO2013097073A1 - Stream processing method and apparatus

Info

Publication number: WO2013097073A1
Application number: PCT/CN2011/084643
Authority: WO
Inventors: 刘晓
Original assignee: 华为技术有限公司
Priority date: 2011-12-26
Filing date: 2011-12-26
Publication date: 2013-07-04
Also published as: CN103282880A; CN103282880B

Abstract

Disclosed are a stream processing method and apparatus. The method comprises receiving an event belonging to a current stream processing window; caching the event belonging to the current stream processing window; judging whether the cached event belonging to the current stream processing window meets a first expression associated with a first key word; in the case of a judgment result of not meeting the first expression, not deleting the cached event belonging to the current stream processing window, and receiving a next event belonging to the current stream processing window; and in the case of a judgment result of meeting the first expression, outputting the cached event belonging to the current stream processing window according to the first key word, wherein the first key word is used for indicating outputting of the cached event belonging to the current stream processing window and meeting the first expression. By adopting the present invention, the output of all the events meeting the conditions can be realized without nesting of multiple rules.

Description

Technical field

The present invention relates to the field of complex event processing, and in particular, to a stream processing method and apparatus. Background technique

At present, the amount of data that the information processing system needs to process is getting larger and larger, and the real-time requirements are getting higher and higher. Complex Event Processing (CEP) technology is an emerging event stream-based data processing technology. It treats system data as different types of event streams. By analyzing the relationship between events, detecting and establishing associations between different events, and using techniques such as filtering, association, and aggregation, finally generating and triggering advanced events by simple events. Or business process. Unlike traditional database technology, CEP technology does not process large amounts of data stored statically, but processes dynamically generated, time-series data streams in real time to discover relationships between a large number of different events—patterns, exceptions, Missing, hierarchy, etc.

In the CEP system, when an event flow arrives, the query statement continuously acts/executes (ie, performs filtering, association, etc.) on the event. Since the event is continuously passed through the CEP system, in CEP processing, it is often necessary to define a window, and the query is generally calculated based on the specific window. For example: For stock event streams, count the average of all stocks in the last minute; or count the stock averages for the last 100 trades.

There are many ways to describe a query in a CEP system. The Continuous Query Language (CQL) is a better description language. CQL is a SQL-based description language that allows continuous querying of streaming data. EPL (Event Process Language) is a stream processing description language of Structured Query Language (SQL). It is based on SQL description language and describes the description of window, mode and so on.

In the EPL language, the stream name is followed by the FROM clause (assuming the stream name is S), and the definition after the stream name "." indicates the concept of the window. For example: Select * From S. win: time (60 sec), which means that 60 seconds of stream processing window is opened for stream S. In the EPL, there are mainly the following windows: time sliding window (represented as S. win: time (60 sec)), number sliding window (represented as S. win: length (60 sec)), grouping window (represented as S. std: groupwin(*) ) and so on.

In the EPL language, having **>=10 means that only the return condition is satisfied in the query result. As a result, with SQL usage class 4; on myevent(CallCounter>= 10) as a update mywindow as b set flag=true where a.strlmsi=b.strlmsi and b.flag=false means that when the where condition is met, myevent flow When the field CallCounter>=10 satisfies a certain condition, the content of the flag field in mywindow is updated.

In the prior art, the existing keyword Having in the stream processing can only output the record after the condition is satisfied, and the previous data is considered to be the condition data not processed together with the expired data, and cannot be output together with the record satisfying the condition. , such as: The condition is having > 5; then if there are 10 records that actually satisfy the condition, the actual output is only 5~10.

If all the records satisfying the condition are to be output, in the prior art, a multi-rule nesting implementation is required: two copies of the data in the same period are generated, B, the user who satisfies the condition is found from A, and then from B. , find out the call history of this user. Summary of the invention

A technical problem to be solved by embodiments of the present invention is to provide a stream processing method and apparatus. The output of all events that satisfy the condition can be implemented without multi-rule nesting.

In order to solve the above technical problem, in one aspect, an embodiment of the present invention provides a stream processing method, where the method includes:

Receiving an event belonging to the current stream processing window;

Caching the events belonging to the current stream processing window;

Determining whether the cached event belonging to the current stream processing window satisfies the first expression related to the first keyword;

When the judgment result is that the first expression is not satisfied, the cached event belonging to the current stream processing window is not deleted, and the next event belonging to the current stream processing window is received;

When the result of the determination is that the first expression is satisfied, the cached event belonging to the current stream processing window is output according to the first keyword, where the first keyword is used to indicate that the output buffered belongs to the current stream processing. The window and the event of the first expression is satisfied.

In another aspect, the embodiment of the present invention further provides a stream processing apparatus, where the apparatus includes: a receiving unit, configured to receive an event belonging to a current stream processing window;

a cache unit, configured to cache the event that belongs to the current stream processing window;

a first expression determining unit, configured to determine whether the cached event belonging to the current stream processing window is Satisfying the first expression related to the first keyword;

And an output unit, configured to: when the first expression determining unit determines that the first expression is satisfied, output the cached event belonging to the current stream processing window according to the first keyword, where the first keyword is used Instructing to output an event that belongs to the current stream processing window and satisfying the first expression; a loop unit, configured to not delete the cached when the first expression determining unit determines that the first expression is not satisfied The event belonging to the current stream processing window, and triggering the receiving unit to receive the next event belonging to the current stream processing window.

In the embodiment of the present invention, when processing the first expression related to the first keyword, even if the first expression is not satisfied, the cached event belonging to the current stream processing window is not deleted, and the current stream is not When the event of the processing window does not satisfy the condition, the events are not retained, so that at the last output, all events in the stream processing window that cause the condition are satisfied can be output, and not only the event after the condition is satisfied. DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any inventive labor.

1 is a schematic flow chart of a stream processing method in an embodiment of the present invention;

2 is a schematic diagram of a composition of a stream processing device in an embodiment of the present invention;

3 is a schematic diagram of a composition of a cache unit in an embodiment of the present invention;

4 is a schematic diagram of another composition of a stream processing apparatus in an embodiment of the present invention;

FIG. 5 is another schematic flowchart of a stream processing method in an embodiment of the present invention; FIG.

6 is a schematic flowchart of processing a plurality of events by using the stream processing method in FIG. 5;

7 is a schematic diagram showing changes in events included in a sliding window when the sliding window is slid in multiple times in the embodiment of the present invention;

FIG. 8 is a schematic flowchart of processing a plurality of events by using the stream processing method in the embodiment of the present invention; FIG. 9 is another schematic diagram of event changes included when the sliding window is slid multiple times in the embodiment of the present invention. detailed description

BRIEF DESCRIPTION OF THE DRAWINGS The technical solutions in the embodiments of the present invention will be described in detail with reference to the accompanying drawings. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without creative work are within the scope of the present invention.

In the current SQL rule description of the stream processing software, the having keyword can only retain the data after the condition is met in the period (ie, the stream processing window). In the embodiment of the present invention, by adding the first keyword (for example, the had keyword), all data in the period that causes the condition to be reached can be retained. And had can be followed by a complex conditional expression, which can be a single-flow conditional expression or a multi-stream conditional expression.

As shown in FIG. 1, it is a schematic flowchart of an event stream-based data processing method in an embodiment of the present invention. The process includes the following steps:

101. Receive an event belonging to a current stream processing window. In stream processing, events are entered as event streams. The event can include a call log, a short message send record, or a website click record.

102. Cache an event belonging to the current stream processing window. Specifically, the event belonging to the current stream processing window may be cached in the memory of the stream processing engine.

When the event is cached, the second expression may be further cached, that is, whether the event belonging to the current stream processing window satisfies the second expression related to the second keyword; if the judgment result is that the second expression is not satisfied And deleting the cached event belonging to the current stream processing window, and buffering the received event belonging to the current stream processing window; otherwise, buffering the received event belonging to the current stream processing window.

Of course, the expression may also be a third expression, that is, determining whether to cache the event of the current stream processing window according to the third expression, and when the judgment result is yes, buffering the event belonging to the current stream processing window.

103. Determine whether the cached event belonging to the current stream processing window satisfies the first expression related to the first keyword.

The first keyword may be a had keyword.

104. When the result of the determination is that the first expression is not satisfied, the cached event belonging to the current stream processing window is not deleted, and the next event belonging to the current stream processing window is received.

105. When the result of the determination is that the first expression is satisfied, outputting a cached event belonging to the current stream processing window according to the first keyword, where the first keyword is used to indicate that the output is cached An event belonging to the current stream processing window and satisfying the first expression.

Further, in the stream processing process in the embodiment of the present invention, the stream processing window may be a time window that changes with time, such as a sliding time window, a skip time window, etc., when the current stream processing window moves, the cache is deleted. Events that are not part of the moved stream processing window.

At the same time, when it is determined whether the cached event belonging to the current stream processing window satisfies the first expression related to the first keyword, the intermediate calculation result of processing the event of the current stream processing window is cached; When the word output buffer belongs to the event of the current stream processing window, the final calculation result obtained based on the intermediate calculation result is output.

The above method can be used in a complex event processing CEP system using a CQL language, an EPL language, or a CEP rule description language like SQL.

Correspondingly, the embodiment of the present invention further provides a stream processing apparatus. As shown in FIG. 2, the apparatus includes: a receiving unit 10, configured to receive an event belonging to a current stream processing window; and a buffer unit 20, configured to cache the An event belonging to the current stream processing window; the first expression determining unit 30 is configured to determine whether the cached event belonging to the current stream processing window satisfies a first expression related to the first keyword; the output unit 40 is configured to When the judgment result of the first expression judging unit is that the first expression is satisfied, the cached event belonging to the current stream processing window is output according to the first keyword, wherein the first keyword is used to indicate that the output is cached. An event that belongs to the current stream processing window and satisfies the first expression; the loop unit 50 is configured to: when the first expression determining unit determines that the first expression is not satisfied, does not delete the cached current stream processing An event of the window, and triggering the receiving unit to receive a next event belonging to the current stream processing window.

The cache unit 20 may be specifically configured to cache the events belonging to the current stream processing window into the memory of the stream processing engine.

As shown in FIG. 3, the cache unit 20 may include: a second expression determining subunit 200, configured to determine whether the event belonging to the current stream processing window satisfies a second expression related to the second keyword; 202, configured to: when the second expression determines that the subunit determines that the second expression is not satisfied, deletes the cached event that belongs to the current stream processing window, and buffers the received event that belongs to the current stream processing window; Otherwise, the received event belonging to the current stream processing window is cached. Or: the cache unit includes: a third expression determining subunit (not shown), configured to determine, according to the third expression, whether to cache an event of the current stream processing window, and when the judgment result is yes, buffering the belonging The event of the current stream processing window. Further, as shown in FIG. 4, the foregoing apparatus may further include: a processing unit 60, configured to: when determining whether the cached event belonging to the current stream processing window satisfies the first expression related to the first keyword, An intermediate calculation result of processing an event of the current stream processing window; the output unit 40 is further configured to output, when the event belonging to the current stream processing window is cached according to the first keyword, output a final result obtained according to the intermediate calculation result Calculation results.

The event described above may include a call record, a short message transmission record or a website click record, the first keyword is a had keyword, the device is used in a complex event processing CEP system, and the CEP system uses a CQL language. , EPL language or CEP rule description language like SQL. For other terms and details in the embodiment of the device, reference may be made to the details of the foregoing or later method embodiments, which are not described herein.

In the embodiment of the present invention, when processing the first expression related to the first keyword, even if the first expression is not satisfied, the cached event belonging to the current stream processing window is not deleted, and the current stream is not When the event of the processing window does not satisfy the condition, the events are not retained, so that at the last output, all events in the stream processing window that cause the condition are satisfied can be output, and not only the event after the condition is satisfied.

In a specific embodiment of the present invention, when the event is a call record, all call records satisfying the condition may be output according to the first expression related to the first keyword, for example, finding a malicious call generated by the user, and then according to these The call record is recorded in the database to obtain a specific bill; it is not because the part of the call record before the condition is satisfied is not cached, and the last acquired bill is incomplete.

As shown in FIG. 5, it is another specific flowchart of the stream processing method in the embodiment of the present invention. In this example, the event is processed according to the first expression and the second expression.

201. Receive an event belonging to a current stream processing window.

202. Determine whether the event belonging to the current stream processing window satisfies a second expression related to the second keyword. If the result of the judgment is that the second expression is not satisfied, 203 is performed; if the result of the judgment is that the second expression is satisfied, 204 is performed.

203. Delete an event that is cached in the memory of the stream processing engine and belongs to the current stream processing window, and cache the received event belonging to the current stream processing window into a memory of the stream processing engine. Then execute 205

204. Cache the received event belonging to the current stream processing window into a memory of the stream processing engine. Then, go to step 205. 205. Determine whether the cached event belonging to the current stream processing window satisfies the first expression related to the first keyword. When the result of the determination is that the first expression is not satisfied, execution 206; when the result of the determination is that the first expression is satisfied, 207 is performed.

206. When the result of the determination is that the first expression is not satisfied, the cached event belonging to the current stream processing window is not deleted, and returning to step 201 to receive the next event belonging to the current stream processing window.

207. When the result of the determination is that the first expression is satisfied, the cached event belonging to the current stream processing window is output according to the first keyword.

The process can be ended after the execution of this step ends, or return to step 201 to continue receiving the next event.

On the basis of the foregoing embodiment, the flow processing window in a specific embodiment may be a one-minute sliding window, and the first expression is that the number of consecutive calls of the user in the one-minute sliding window is greater than or equal to five times; the second expression The interval between adjacent calls is less than 5 seconds (At < 5s ), and the second keyword is to record only the call records that satisfy the second expression. The rules described by the EPL are as follows:

Select * from stream.win:time(60 sec) group by a had count(*)>=5 where (timestamp - previous (times tamp ) ) ( 5 s

Where, time (timestamp - previous ( timestamp ) ) < 5 s means that the interval between two adjacent events is less than 5 seconds. Then, as shown in FIG. 6, the processing flow of the event flow for the image is:

301. Receiving event 1, since event 1 is the first event, it is not necessary to determine the interval between the event and the previous event, and the event 1 is cached. At this time, the event count in window 1 is 1 (and 1<5), then Cached event 1 is not deleted, but it is not output.

302. Receive event 2, and determine that the interval between event 1 and event 2 is less than 5 seconds, then cache event 2 (ie: two events belonging to window 1 that have been cached at this time are event 1 and event 2, respectively. ), when the event count in window 1 is 2 (and there is 2<5), the cached events 1 and 2 are not deleted, but they are not output.

303. Receive event 3, and determine that the interval between event 2 and event 3 is greater than 5 seconds, then delete the cached event 1 and event 2, and only cache event 3, and the event count in window 1 is 1 (And there is 1<5), the cached event 3 is not deleted, but it is not output.

304, and when the sliding time of the window is up, slide the window. When window 1 slides to window 2 (that is, the current stream processing window changes from window 1 to window 2), the event that slides out of the window (here, event 1) is deleted, and then subsequent events 4 to 6 are received and processed. At the same time, the count changes accordingly, and according to the above The law continues with event processing.

305, and when the sliding time of the window is up, slide the window. When window 2 slides to window 3 (ie: the current stream processing window changes from window 2 to window 3), the event that slides out of the window (here, event 2) is deleted, event 7 is received, and event 6 is judged at this time. The interval between events 7 is less than 5 seconds, then the cached event is 3~7, and the event count in the window is 5 (and 5=5), then the cached events 3~7 are not deleted. And output the cached events 3~7.

306. Continue to process subsequent events in a manner similar to that described above.

Of course, the above sliding time window can also be a jump time window, and the processing flow is similar.

As shown in FIG. 8, the flow is processed according to the first expression and the third expression. In this example, the first keyword is the had keyword, the expression for the first keyword is count ( ) > 5 , and the third expression is ** ! =null, ie the field ** is not empty. For the event flow shown in Figure 9, the processing flow is:

401. Receive event 1, because event 1 is the first event, determine that its field ** is not empty, and cache the event. At this time, the event count in window 1 is 1 (and there is 1<5), then the deletion is not deleted. Cache event 1 but not output.

402. Receive event 2, determine that its field ** is not empty, and cache the event. At this time, the event count in window 1 is 2 (and there is 2<5), then the cached events 1 and 2 are not deleted, but also Do not output. The same thing is handled by events 3~5. When the sliding time of the window is up, slide the window. When window 1 is slid into window 2, the event that slides out of the window is deleted, and the count is changed accordingly, and the event processing is continued as described above.

403. Receive event 6, and judge that the field ** is not empty, then the cached event is 2~6, and the event count in the window is 5 (and 5=5), then the cached is not deleted. Event, and output cached events 2~6.

404. Continue to process subsequent events by using a method similar to the above.

In the above embodiment, the keyword had can also be used in combination with various other window definitions, query conditions, schema definitions, etc., such as Count ( a ), count( [all | distinct] expression [, filter_expr]) \ count(* [, filter_ expr]) and so on.

According to the description of the foregoing embodiment, in the prior art, in a statistical period: a single rule existing keyword (eg, having ) can only output a record that satisfies the condition, and deletes when a new event is received. Processed events; if you want to list all the records that cause the condition to be met, you must implement multiple rule combinations. In the embodiment of the present invention, in the statistical period: a single rule keyword expansion (eg, increase) Add the keyword had) to list all the records that cause the condition to be met, without the need for multiple rule combinations. That is, in the embodiment of the present invention, the record buffer satisfying the condition of the post-had expression is obtained when the rule definition output condition or the expression condition after the had is not satisfied (for a simple event count, the expression condition does not conform to the had condition. Output, but for complex situations, such as the case of packet statistics, the event is a record of events in different groups. For example, packet A may satisfy 5 times, then the record of packet A is output, and packet B does not satisfy 5 times. , then the record associated with group B has no output), will output all the records that satisfy the condition of the post-had expression, clear the buffer and continue counting.

A person skilled in the art can understand that all or part of the process of implementing the above embodiment method can be completed by a computer program to instruct related hardware, and the program can be stored in a computer readable storage medium. In execution, the flow of an embodiment of the methods as described above may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

The above description is only a preferred embodiment of the present invention, and the scope of the present invention is not limited thereto, and the equivalent changes made by the claims of the present invention are still within the scope of the present invention.

Claims

Rights request

A stream processing method, the method comprising:

Receiving an event belonging to the current stream processing window;

Caching the events belonging to the current stream processing window;

2. The stream processing method according to claim 1, wherein the buffering the event belonging to the current stream processing window specifically includes:

The event belonging to the current stream processing window is cached into the memory of the stream processing engine.

3. The stream processing method according to claim 1 or 2, wherein the buffering the event belonging to the current stream processing window specifically includes:

Determining whether the event belonging to the current stream processing window satisfies a second expression related to the second keyword;

If the judgment result is that the second expression is not satisfied, the cached event belonging to the current stream processing window is deleted, and the received event belonging to the current stream processing window is buffered; otherwise, the buffer received the current stream processing window belongs to event.

The stream processing method according to any one of claims 1 to 3, wherein the method further comprises:

When the current stream processing window moves, an event in the cache that does not belong to the moved stream processing window is deleted.

The stream processing method according to any one of claims 1 to 4, wherein, in the determining whether the event belonging to the current stream processing window of the cache satisfies the first expression step related to the first keyword Afterwards, the method further includes: buffering an intermediate calculation result that processes an event of the current stream processing window;

When the cached event belonging to the current stream processing window is output in accordance with the first keyword, the final calculation result obtained based on the intermediate calculation result is output.

The stream processing method according to claim 1 or 2, wherein the buffering the events belonging to the current stream processing window comprises:

Determining whether to cache the event of the current stream processing window according to the third expression, and when the judgment result is YES, buffering the event belonging to the current stream processing window.

The stream processing method according to any one of claims 1 to 6, wherein the event includes a call record, a short message transmission record, or a website click record.

The stream processing method according to any one of claims 1 to 7, wherein the first keyword is a had keyword.

The stream processing method according to any one of claims 1 to 8, wherein the method is used in a complex event processing CEP system, and the CEP system uses CQL language, EPL language or CEP-like CEP Rule description language.

10. A stream processing device, the device comprising:

a receiving unit, configured to receive an event that belongs to a current stream processing window;

a first expression determining unit, configured to determine whether the cached event belonging to the current stream processing window satisfies a first expression related to the first keyword;

And an output unit, configured to: when the first expression determining unit determines that the first expression is satisfied, output the cached event belonging to the current stream processing window according to the first keyword, where the first keyword is used Instructing to output an event that is cached and belongs to the current stream processing window and satisfies the first expression; a looping unit, configured to: when the first expression determining unit determines that the first expression is not satisfied, does not delete the cached event belonging to the current stream processing window, and triggers the receiving unit to receive the current stream processing The next event of the window.

The device according to claim 10, wherein the cache unit is specifically configured to cache the event belonging to the current stream processing window into a memory of the stream processing engine.

The device according to claim 10 or 11, wherein the cache unit specifically includes:

a second expression determining subunit, configured to determine whether the event belonging to the current stream processing window satisfies a second expression related to the second keyword;

a processing subunit, configured to: when the second expression determines that the subunit determines that the second expression is not satisfied, deletes the cached event belonging to the current stream processing window, and caches the received current window that belongs to the current stream processing window. Event; otherwise, the received event belonging to the current stream processing window is cached.

The apparatus according to any one of claims 10 to 12, wherein the apparatus further comprises:

a processing unit, configured to: when determining whether the cached event belonging to the current stream processing window satisfies the first expression related to the first keyword, buffering an intermediate calculation result of processing the event of the current stream processing window;

The output unit is further configured to output, when the cached event belonging to the current stream processing window is output according to the first keyword, a final calculation result obtained according to the intermediate calculation result.

The device according to claim 10 or 11, wherein the buffer unit specifically includes:

The third expression determining subunit is configured to determine, according to the third expression, whether to cache the event of the current stream processing window, and when the judgment result is yes, cache the event belonging to the current stream processing window.

The device according to any one of claims 10 to 14, wherein the event comprises a call record, a short message transmission record or a website click record, and the first keyword is a had keyword. The apparatus is used in a complex event processing CEP system that uses a CQL language, an EPL language, or a CEP rule description language like SQL.