CN110674174B

CN110674174B - Data real-time processing method and data real-time processing system

Info

Publication number: CN110674174B
Application number: CN201910907434.2A
Authority: CN
Inventors: 焦悦光; 胡宗星; 郭璐
Original assignee: Beijing Zetyun Tech Co ltd
Current assignee: Beijing Zetyun Tech Co ltd
Priority date: 2019-09-24
Filing date: 2019-09-24
Publication date: 2020-09-01
Anticipated expiration: 2039-09-24
Also published as: CN110674174A

Abstract

The invention provides a real-time data processing method and a real-time data processing system, which comprise the following steps: displaying a user interface, receiving a first input, and generating an operator of a task of streaming data according to the first input; the operator comprises an application rule operator, and the application rule operator comprises rules in at least one rule set; constructing a task of streaming data according to the operator of the task; and running the task of the streaming data. The data real-time processing system in the embodiment of the invention can process real-time streaming data, thereby greatly improving the data processing efficiency.

Description

Data real-time processing method and data real-time processing system

Technical Field

The invention relates to the technical field of information, in particular to a real-time data processing method and a real-time data processing system.

Background

With the rapid expansion of data, big data processing technology has been developed rapidly. Currently, a big data processing platform can be divided into off-line computation and real-time computation according to a computation mode. With the continuous improvement of the real-time degree of the economic and social information, people have higher and higher requirements on data real-time calculation. For example: real-time calculation is needed in scenes such as risk control systems (wind control systems, compliance checks and the like) for preventing fraud and judging whether funds flow into illegal ways, data processing systems for data extraction, conversion and loading (ETL for short), and the like.

However, the existing rule engine and the operator of the data real-time processing system are executed separately, and the existing rule engine can only process offline data, so that the data processing efficiency is low.

Disclosure of Invention

The embodiment of the invention provides a data real-time processing method and a data real-time processing system, which solve the problems that the conventional rule engine can only process offline data and has low data processing efficiency.

In order to solve the above technical problem, an embodiment of the present invention provides a real-time data processing method, where the method includes:

displaying a user interface, receiving a first input, and generating an operator of a task of streaming data according to the first input; the operator comprises an application rule operator, and the application rule operator comprises rules in at least one rule set;

constructing a task of streaming data according to the operator of the task;

and running the task of the streaming data.

Preferably, in the above method, before the step of displaying the user interface and receiving a first input, and generating an operator of a task of the streaming data according to the first input, the method further includes:

displaying a rule creation interface;

receiving a second input, and creating the rule set according to the second input;

wherein the set of rules contains at least one rule.

Preferably, in the above method, the method further comprises: based on the service scene of the application rule operator, extracting the same content in the multiple rules as a rule template, and storing the rule template in a template database.

Preferably, in the above method, the method further comprises:

inquiring the template database to obtain a list of the rule templates;

selecting a target rule template from the list;

and generating a corresponding rule according to the target rule template.

Preferably, in the above method, the task further includes: operator and operator connection relation.

Preferably, in the above method, the rule includes at least one of: wizard rules, rule sets, score cards, decision tables, decision trees, decision flows, and models.

Preferably, in the above method, the decision flow is composed of execution nodes organized into a directed graph structure, and the execution nodes include: sub-rule node, routing node, start node.

Preferably, in the above method, a subsequent execution node of the routing node is determined according to a subsequent node calculation algorithm configured by the routing node, and the routing node can end the loop execution of the decision flow when a specified condition is satisfied.

Preferably, in the above method, the step of running the task of the streaming data includes:

receiving streaming data;

calculating the flow data by using an application rule operator in the operators to generate an output event;

and outputting and displaying the output event.

Preferably, in the above method, the application rule operator comprises a rule engine;

the step of calculating the stream data by using an application rule operator in the operators to generate an output event includes:

reading the attribute of the event corresponding to the stream data to a rule variable in the rule engine;

a rule engine in the application rule operator executes the rules in the rule set to obtain an output result of the rules; wherein the output result of the rule is taken as the output event.

Preferably, in the above method, the rule variable includes a state variable, and a value of the state variable remains unchanged after one event processing is completed and can be accessed when the next event processing is performed.

Preferably, in the above method, the model includes a machine learning model, and the machine learning model includes an algorithm and related parameters.

The embodiment of the present invention further provides a real-time data processing system, where the real-time data processing system includes:

the human-computer interface display module is used for displaying a user interface, receiving first input and generating an operator of a task of streaming data according to the first input; the operator comprises an application rule operator, and the application rule operator comprises rules in at least one rule set;

the task construction module is used for constructing a task of the streaming data according to the operator of the task;

and the running module is used for running the task of the streaming data.

Preferably, the data real-time processing system further includes:

the rule interface module is used for displaying a rule creating interface;

the rule creating module is used for receiving second input and creating the rule set according to the second input;

wherein the set of rules contains at least one rule.

Preferably, the data real-time processing system further includes:

and the extraction module is used for extracting the same content in the multiple rules into a rule template based on the service scene of the application rule operator, and storing the rule template in a template database.

Preferably, the data real-time processing system further includes:

the query module is used for querying the template database and acquiring the list of the rule templates;

a selection module for selecting a target rule template from the list;

and the generating module is used for generating a corresponding rule according to the target rule template.

Preferably, in the data real-time processing system, the task further includes: operator and operator connection relation.

Preferably, in the above data real-time processing system, the rule includes at least one of: wizard rules, rule sets, score cards, decision tables, decision trees, decision flows, and models.

Preferably, in the data real-time processing system, the decision flow is composed of execution nodes organized into a directed graph structure, and the execution nodes include: sub-rule node, routing node, start node.

Preferably, in the data real-time processing system, a subsequent execution node of the routing node is determined according to a subsequent node calculation algorithm configured by the routing node, and the routing node can end the loop execution of the decision flow when a specified condition is satisfied.

Preferably, in the data real-time processing system, the operation module includes:

the receiving submodule is used for receiving the streaming data;

the calculation submodule is used for calculating the flow data by using an application rule operator in the operators to generate an output event;

and the output submodule is used for outputting and displaying the output event.

Preferably, in the data real-time processing system, the application rule operator includes a rule engine;

the calculation submodule is specifically configured to:

the rule engine in the application rule operator executes the rules in the rule set to obtain the output result of the rules; wherein the output result of the rule is taken as the output event.

Preferably, in the above data real-time processing system, the rule variable includes a state variable, and a value of the state variable remains unchanged after one event processing is completed and can be accessed when the next event processing is performed.

Preferably, in the above data real-time processing system, the model includes a machine learning model, and the machine learning model includes an algorithm and related parameters.

The embodiment of the invention also provides a data real-time processing system, which comprises a processor, a memory and a computer program which is stored on the memory and can run on the processor, wherein when the computer program is executed by the processor, the steps of the data real-time processing method are realized.

The embodiment of the invention also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the data real-time processing method are realized.

The invention provides a real-time data processing method and a real-time data processing system, which comprise the following steps:

displaying a user interface, receiving a first input, and generating an operator of a task of streaming data according to the first input; the operator comprises an application rule operator, and the application rule operator comprises rules in at least one rule set; constructing a task of streaming data according to the operator of the task; and running the task of the streaming data. In the embodiment of the invention, the rule engine is placed in the operator of the data real-time processing system, so that the rule engine can process real-time stream data, and the data processing efficiency is greatly improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive exercise.

FIG. 1 is a flow chart of a method for real-time processing of data according to an embodiment of the present invention;

FIG. 2 is a flowchart of a data real-time processing method step 103 according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating a relationship between a rule engine and an application rule operator in the data real-time processing method according to an embodiment of the present invention;

FIG. 4 is a flow chart of a method for real-time processing of data according to an embodiment of the present invention;

FIG. 5 is a flow of credit card application decisions provided by an embodiment of the present invention;

FIG. 6 is a decision flow in an anti-fraud real-time task and application rule operator provided by an embodiment of the present invention;

FIG. 7 is a block diagram of a data real-time processing system provided by an embodiment of the present invention;

fig. 8 is a block diagram of an operation module of the data real-time processing system according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a flowchart illustrating a real-time data processing method according to an embodiment of the invention. The method can be applied to a data real-time processing system, as shown in fig. 1, and comprises the following steps:

step 101, displaying a user interface, receiving a first input, and generating an operator of a task of streaming data according to the first input; the operator comprises an apply rules operator comprising rules in at least one rule set.

Wherein the task further comprises: operator and operator connection relation.

Here, the user interface is exemplified by a graphic interface, and the data real-time processing system displays the user interface for creating a streaming data task. Wherein the first input may include, but is not limited to: editing the configuration information of the streaming data task, adding and deleting operators, editing the configuration information of the operators, drawing connecting lines among the operators, editing the configuration information of the connecting lines and the like. The operators include application rule operators, and may also include other operators, such as input operators, output operators, and the like. Wherein the application rule operator comprises at least one rule in a rule set, and the rule in the rule set is read and executed by a rule engine. Rules may be compiled prior to execution to convert to a form that facilitates increased execution speed.

102, constructing a task of streaming data according to the operator of the task;

and constructing a complete task of the stream data according to the operator in the task and the connection relation between the operator and the operator in the task.

And 103, running the task of the streaming data.

And after the task of the stream data is constructed, executing the task of the stream data.

Optionally, as shown in fig. 2, step 103 specifically includes the following steps:

step 1031, receiving the streaming data.

Here, the stream data generally refers to data continuously generated by a data source, and the generation of the data is usually not explicitly terminated, so that the data cannot be processed until all the data is accessible, and the generated data is usually processed in real time. The data real-time processing system receives the stream data for processing.

And 1032, calculating the flow data by using an application rule operator in the operators to generate an output event.

Optionally, the application rule operator comprises a rule engine. Or, the application rule operator invokes the rule engine. The rule engine performs calculations on given input data according to rules defined in advance and outputs the results of the calculations, i.e., the rule engine is actually the executor of the rules.

Step 1032 specifically includes:

as shown in fig. 3, reading the attribute of the event corresponding to the stream data to the rule variable in the rule engine.

The rule variables include input variables, output variables, intermediate variables, state variables, and the like.

Here, the stream data corresponds to different events according to different application scenarios. For example, in a risk control system (a wind control system, a compliance check, etc. for anti-fraud and determining whether funds flow into illegal routes), in this application scenario, an event corresponding to streaming data is a transaction record of a user.

Here, the attribute of the event corresponding to the stream data includes each field of the event corresponding to the stream data. And reading each field of the event corresponding to the stream data to an input variable in the rule engine. The input variables refer to fields in the input data of the rule engine, wherein the rule engine reads and executes the rules in the rule application operators, and the input variables are actually identical to the fields in the input data of the rule application operators.

Here, a rule engine in the application rule operator executes the rules in the rule set to obtain output variables corresponding to the rules.

Optionally, the rule variable includes a state variable, and a value of the state variable remains unchanged after one event processing is completed and can be accessed when the next event processing is performed.

Here, for the data real-time processing system, the event entering the data real-time processing system is sequential, and the output result of a single event entering the data real-time processing system is not only related to the event itself, but also related to the event entering the data real-time processing system before, which can be realized by the state variable. The state variable may be used to keep the value obtained after the end of the previous event processing until the next event processing, i.e. data may be passed in the preceding and succeeding event processing. The rule may refer to a plurality of state variables, and the values of the state variables are not limited to numeric values or character strings, but may also be various complex types such as arrays, sets, key-value pairs, and their mutual nesting, which is not specifically limited in this embodiment of the present invention.

The input variable in the rule engine is read from the corresponding attribute of the input event, the output variable in the rule engine is written into the corresponding attribute of the output event, and the state variable in the rule engine is stored in the state variable corresponding to the application rule operator, so that the state variable corresponding to the operator can be read or written. Correspondence here refers to a one-to-one mapping from the variables of the rules engine to the event attributes and operator state variables according to a given method, for example, the same name may be called a mapping method. As shown in fig. 3, if the rule engine execution rule includes a state variable E and a state variable F, the state variable E and the state variable F are written into an operator state variable E and an operator state variable F corresponding to the application rule operator, respectively. In addition, the rule engine may also read the corresponding operator state variables from the application rule operator to the state variables of the rule engine.

And step 1033, outputting and displaying the output event.

And writing the output variable of the rule engine into the attribute corresponding to the output event, and then displaying the attribute corresponding to the output event.

According to the data real-time processing method provided by the embodiment of the invention, the rule engine is placed in the operator of the data real-time processing system and is used for processing the streaming data in real time, so that the data processing efficiency and the real-time performance are greatly improved.

Referring to fig. 4, before the step 101, the data real-time processing method further includes:

step 104, displaying a rule creation interface.

Here, the rules may be created, edited, and managed through a graphical user interface.

It should be noted that, besides the graphical user interface, the rules can be created, edited and managed by inputting codes. The created, edited and managed rules may be stored and transmitted in a given format, may be exported from the system, or may be imported into the system. The embodiments of the present invention are merely exemplary, and are not particularly limited thereto.

Step 105, receiving a second input, and creating the rule set according to the second input;

wherein the set of rules contains at least one rule.

Here, the input of the user on the rule creation interface may be received, and the input may be a variable input by the user, or a selection operation input by the user according to an option of the rule creation interface, or the like. And creating at least one rule in a rule creating interface according to the input of the user to form a rule set. In the rule creation interface, the construction and management of rule variables and constants can be performed, and the rule variables comprise input variables, output variables, intermediate variables, state variables and the like.

The input variables refer to fields in the input data of the rule engine, and the fields in the input data of the operators in the data real-time processing system are equivalent to the fields in the input data of the operators in the data real-time processing system when the input variables are applied to stream data processing. Wherein the input data is a value of an input variable.

The output variables refer to each field in the output data of the rule engine, and are equal to each field in the output data of an operator in the data real-time processing system when the output variables are applied to stream data processing.

Intermediate variables (i.e., parameters) are used for passing data between individual sub-rules in a rule, such as a decision flow or rule set, and are input or output for individual sub-rules, but are neither input nor output for the rule as a whole, such as a decision flow or rule set.

The constant is data defined by a user in advance, and the value is fixed and unchanged (does not change along with input data), so that the constant is convenient for the user to use in a selection mode when defining the rule.

Optionally, at least one rule in the rule set contains at least one state variable.

A state variable is a variable whose value can be saved to the next event processing epoch after one event processing is completed.

In the prior art, the execution of a rule has no state variables. In the invention, the state variables are introduced when the rules are created and the rule variables are constructed, so that the rule variables of at least one rule contain the state variables, the rule engine processes the stream data in the application rule operator of the data real-time processing system, and when the events corresponding to the stream data contain the states, the state variables are required to be used and can be stored to the next event processing period.

Optionally, the rule includes at least one of: wizard rules, rule sets, score cards, decision tables, decision trees, decision flows, and models.

Optionally, the decision flow is composed of execution nodes organized into a directed graph structure, where the execution nodes include: sub-rule node, routing node, start node.

Optionally, a subsequent execution node of the routing node is determined according to a subsequent node calculation algorithm configured by the routing node, and the routing node can end the loop execution of the decision flow when a specified condition is satisfied.

Optionally, the model comprises a machine learning model, and the machine learning model comprises an algorithm and related parameters.

The specific descriptions for the above types of rules are as follows:

a guide rule: the device comprises a condition part, an action part when a condition is established, and an action part when an optional condition is not established. The method for executing the guided rule is as follows: when the input data enables the condition to be satisfied, executing the action when the condition is satisfied; when the input data causes the condition to be not satisfied, the action or no action when the condition is not satisfied is executed. A directed rule may be a sub-rule in a rule set.

A scoring card: the method is composed of a plurality of scoring items and a scoring summarizing mode. Each scoring item is composed of mutually exclusive (cannot be satisfied simultaneously) sets of conditions and a score corresponding to each condition. The method of scoring card execution is: for each scoring item, when the input data enables a certain condition to be satisfied, the score of the scoring item is the score corresponding to the condition. And finally, summarizing the scores of all the scoring items according to a specified scoring summarizing mode (such as summation, average, weighted average and the like), and taking the obtained final score as output.

Decision table: consisting of mutually exclusive (not being able to be satisfied simultaneously) sets of conditions and actions corresponding to each condition. The method executed by the decision table is as follows: when the input data is such that any one of the plurality of conditions is satisfied, an operation corresponding to the condition is executed.

Decision tree: consisting of a plurality of conditions organized into a tree structure and actions located on leaf nodes. The method for executing the decision tree is as follows: and calculating all conditions on the child nodes of the current node in sequence from the root node, switching to the node and continuing to judge in sequence if the conditions on a certain node are satisfied by inputting data, repeating the above processes until a certain leaf node is reached, and then executing the action on the leaf node.

A rule set: is composed of a plurality of ordered sub-rules. The sub-rules may be any type of rule, including rule sets, and thus, rule sets may be nested. The method for executing the rule set comprises the following steps: each sub-rule is executed in order.

And (4) decision flow: consisting of execution nodes organized into a directed graph structure. The execution node includes: the sub-rule node and the start node further comprise a routing node. The sub-rules may be any type of rule, including decision flows, and thus, decision flows may be nested. The method for executing the decision flow comprises the following steps: from the starting node, its successor nodes are executed. When a node completes its execution, its subsequent nodes are executed, and the above process is repeated until there are no subsequent nodes. Wherein, the subsequent nodes of the starting node and the sub-rule node are fixed and can be obtained from the graph, and the subsequent nodes of the routing node are dynamically decided according to the configured subsequent node calculation algorithm. The graph of the decision flow can have loops, so that the loop execution can be realized, and the loop execution of the decision flow can be skipped/ended when the specified condition is met by combining the setting of the routing node.

Model: the model of the specified format provided by the system or the external system. The model execution method comprises the following steps: the input data is calculated according to the designated model and an output result is generated. The model may be a machine learning model, including algorithms and associated parameters.

Wherein each decision flow may comprise at least one of: one or more rule sets, one or more score cards, one or more decision tables, one or more decision trees, one or more models; the decision flow can be circulated and nested. The rule set may include a plurality of sub-rules, each of which may be any of the above types of rules, and the rule set may be round-robin, nested. The guidance rules, scoring cards, decision tables, decision trees, models cannot be decomposed into smaller individuals.

The machine learning model is a machine learning model which is trained in a machine learning platform of the data real-time processing system or a machine learning platform of other systems (external systems), is imported into the rule engine, can be used for establishing a decision flow and can be used for model management. The management of the invention comprises editing, deleting and other operations. And running the decision flow, inputting variables, and outputting a result according to the established decision flow.

Optionally, the data real-time processing method further includes:

based on the service scene of the application rule operator, extracting the same content in the multiple rules as a rule template, and storing the rule template in a template database.

And querying the template database to obtain the list of the rule templates.

A target rule template is selected from the list.

And generating a corresponding rule according to the target rule template.

Specifically, in a certain type of application scenario, the design of multiple rules has high repeatability and similarity, and the same part can be extracted as a rule template and stored in a template database, and operations such as query, modification, deletion and the like can be performed. When the system is called subsequently, a user obtains a list of templates by querying the database, selects a proper template from the list, and automatically generates a corresponding rule according to the template. On the basis, the user can adjust input variables, parameters, output variables and the like of guide rules, rule sets, score cards, decision trees, decision streams, models and/or decision tables and the like, and add and delete rules to enable the rules to meet the current actual requirements. After the user uses the rule template, a small amount of necessary settings are needed to generate a new rule. Specifically, some templates may be preset based on the service scenario, that is, the stored templates may be classified based on the service scenario.

The rules engine processes input data according to user-defined rules and generates output data. Decision flows, rule sets, score cards, decision tables, decision trees, models, etc. are all implemented as special types of rules.

The following describes the creation process of the credit card application rule by taking a decision flow as an example.

And setting a starting node and a sub-rule node in the rule creation interface. For example, by dragging, selecting, inputting a code, etc. In addition, a routing node may be provided, and a node such as the routing node may be configured. And setting the connection relation between the nodes to complete the creation of the decision flow. And further, each node can be customized and adjusted. The embodiments of the present invention are merely exemplary, and are not particularly limited thereto.

Specifically, the method comprises the following steps:

first, a rule set, named "filter", is created, as shown in table 1:

serial number	Rule name	Status of state
			1	Whether or not it has applied for	Has been activated
2	Age restriction	Has been activated

Table 1 credit card application rule set: screening

Two wizard rules are created within the rule set: the first is "applied for", see table 2, as table 2: if the credit card of the bank is judged to exist, the application result is refused, the continuous judgment is stopped, and the rule has no 'else' part, so that the rule does not meet the condition, no action is executed, and the judgment is continued; the second is "age restriction", as in table 3, see table 3: if the age is judged to be more than or equal to 18 and less than or equal to 50, the judgment is continued, otherwise, the application result is rejection and the continuous judgment is stopped.

If it is not

And is

Input variable/whether there is a present bank credit card

Is equal to

Is that

Then

Table 2 credit card application rule set rule 1: whether or not it has applied for

If it is not

Then

Continue to use

Otherwise

Table 3 credit card application rule set rule 2: age restriction

Then, a model rule is created, named as 'model score', a logistic regression model imported from an external system is used, the model is a calculation model obtained by using historical customer data and credit conditions as training data through a machine learning training method, the credit score of the model can be calculated according to the customer data, and the training process of the model is finished in the external system in advance.

A decision table, named "quota calculation", is then created, as shown in Table 4.

Table 4 credit card application decision table: quota calculation

Finally, a credit card application decision flow is created by using the rule set "screening", the model rule "model scoring" and the decision table "quota calculation", as shown in fig. 5.

Taking customer data (age 35, gender, major, and whether there is a local credit card no) as an example, assuming that his model score is 80, the steps for performing this decision flow are as follows:

1. execution begins with the "start" node and the next node is the "filter".

"screening" node is a rule set consisting of two rules that will be executed in order, first executing the first rule "applied or not".

"applied" or not is a wizard rule that continues to execute the next rule "age restriction" without any action because the customer data (no credit card in this bank already exists) makes the condition false and the rule has no "else" part.

"age restriction" is a wizard rule, and since the condition is satisfied by the customer data (age 35), the action of "then" is executed, the action is "continue", the execution of the rule set "filtering" is completed, and the next node is "model scoring".

The "model score" node calculates its score 80 based on the customer data, and stores it in the variable "score". And the execution of the model scoring node is completed, and the next node is used for calculating the limit.

The "quota calculation" node is a decision table, and obtains a calculation result (quota 10000) based on the output results (rating) of client data (age 35, gender male, academic, etc.) and "model rating".

Specifically, the regular storage format may be JSON, XML, YAML, or the like, and preferably JSON format. All types of rules can be exported and imported from the rule engine. Further, rules for importing external platforms may be introduced: such as rules in drools format.

In a certain type of application scenario, the design of multiple rules has high repeatability and similarity, and the same part can be extracted as a rule template and stored in a template database. Specifically, some templates may be preset based on the service scene, that is, stored templates are classified based on the service scene, for example, some templates are preset in the service scene such as credit rating and wind control alarm, the whole decision flow applied by the credit card is set as the "credit rating template 1" and stored in the template database, when the user meets the requirement in the credit rating scene again, the system reads and applies the "credit rating template 1" from the database only by calling operation, the system automatically generates a corresponding decision flow, rule set, decision table, model and the like, and the user can quickly meet the new requirement only by appropriately modifying the decision flow, rule set, decision table, model and the like.

Further, the execution state can be checked during the execution of the decision flow. The decision flow may include a routing node, and the routing node is used to determine which node or nodes to operate next, and may be determined according to a condition or may be determined randomly according to a probability. The routing node is arranged, so that the flexibility can be improved, and the circulation can be realized conveniently.

In some applications, it is desirable to distribute the input to a number of different downstream nodes in a certain proportion. For example, in a wind control alarm application, the effect of the old model and the new model needs to be compared, the input needs to be distributed to the old model and the new model according to the ratio of 50%/50% for processing, and then the results are counted and compared. Routing nodes can achieve distribution inputs to subsequent new model and old model nodes that are each 50% random based on the settings.

The following describes the process of establishing a streaming data task and executing the streaming data task by taking a decision flow as an example.

A decision flow is created, and a specific decision flow creation method is as shown in the above embodiments, which will not be described herein again. In particular, a decision flow is created using a set of rules. And creating the rule set in a rule creating interface or importing the rule set from an external system to a data real-time processing system. Then, constructing a streaming data task on a user interface: editing the configuration information of the streaming data task, adding and/or deleting operators, editing the configuration information of the operators, drawing connecting lines among the operators, and editing the configuration information of the connecting lines. The operators comprise an input operator, an application rule operator, an output operator and the like. Specifically, the created decision flow is specified as a rule to be used in an application rule operator.

An input operator of the streaming data task reads data from streaming data, an deserializer converts the data read by the input operator into events, a rule engine in a rule operator is applied to execute corresponding rules to process the events, the serializing operator encodes the processed events into data in a corresponding format as output data, and the output data is output by the output operator.

Specifically, the method comprises the following steps:

first, a rule set called "anti-fraud" is created, as in Table 6, and five wizard rules are created within the rule set:

serial number	Rule name	Status of state
			1	Filtering non-consumption types	Has been activated
2	Handling non-failure state consumption	Has been activated
			3	Handling first consumption failure	Has been activated
4	Cumulative number of failures to consume	Has been activated
			5	Output alarm sign	Has been activated

Table 6 rule set: anti-fraud

The first rule is "filter non-consumed type", as in Table 7. See table 7: if the transaction type is judged to be consumption, continuing, otherwise, stopping the continuous judgment. The second rule is "handle non-failed state consumption", as shown in Table 8: if the transaction state is judged to be failure, continuing, otherwise, the consumption failure times are cleared and further judgment is stopped. The third rule is "handle first consumption failure", see table 9: if the consumption failure times are judged to be 0, setting a variable value of 'first consumption failure time' as the transaction time in the current transaction record, setting a variable value of 'consumption amount' as the amount in the current transaction record, and setting a variable value of 'consumption failure times' as 1; the rule has no "else" part, so the condition is not satisfied and no action is performed, and the judgment is continued. The fourth rule is "cumulative number of failed consumptions", as in table 10, see table 10: if the amount in the current transaction record is judged to be smaller than the value corresponding to the consumption amount variable, and the transaction time in the current transaction record is smaller than or equal to the first consumption failure time plus 3 minutes, setting the consumption amount variable value as the amount in the current transaction record, and setting the consumption failure time variable value as the value plus 1 corresponding to the current consumption failure time variable, otherwise, setting the current transaction as the first failure consumption transaction in the next round of continuous failure consumption transaction statistical process, executing the same operation as the part in processing the first consumption failure, namely setting the first consumption failure time variable value as the transaction time in the current transaction record, setting the consumption amount variable value as the amount in the current transaction record, and setting the consumption failure time variable value as 1. The fifth rule is "output alarm flag", as in table 11, see table 11: if the consumption failure times variable value is judged to be 3, then the following steps are executed in sequence:

firstly, setting an alarm flag to be on, and secondly, setting a consumption failure frequency variable value to be 0.

If it is not

And is

Input variables/types

Is equal to

Consumption of

Then

Continue to use

Otherwise

Pause

Table 7 anti-fraud rule 1: filtering non-consumption types

If it is not

And is

Input variables/transaction status

Is equal to

Failure of

Then

Continue to use

Otherwise

Table 8 anti-fraud rule 2: handling non-failure state consumption

If it is not

And is

State variable/number of consumption failures

Is equal to

0

Then

Table 9 anti-fraud rule 3: handling first consumption failure

When "consumption failure times" equal to 0 is true, the currently processed event is the first event satisfying the condition "consumption fails 3 consecutive times", and subsequent processing is not required after the defined action is performed, so "abort". The term "abort" means that the processing of the current event is ended, the stream data processing is not stopped, and the data real-time processing system starts to receive the next event.

If it is not

Then

Otherwise

Table 10 anti-fraud rule 4: cumulative number of failures to consume

If it is not

And is

State variable/number of consumption failures

Is equal to

3

Then

Is provided with	Output variable/alarm flag	Is composed of	Is opened
				Is provided with	State variable/number of consumption failures	Is composed of	0

Table 11 anti-fraud rule 5: output alarm sign

An "anti-fraud" decision stream is then created using the "anti-fraud" rule set, as shown in the inner part of the right-hand dashed box in fig. 6.

The anti-fraud real-time task inputs the transaction record of the user and outputs an alarm mark, and the implemented anti-fraud rule is that 'the consumption is failed for 3 times continuously within 3 minutes and the alarm is triggered if the amount of money is decreased progressively'.

Specifically, the method comprises the following steps:

the task of creating a real-time streaming data, named "anti-fraud", as shown in table 12, contains five operators, named and functional as shown in table 12.

TABLE 12 anti-cheat real-time task operator

In the above example, the data input and output systems may be distributed message pipes Kafka, HTTP services, Socket connections, various databases, file systems, distributed file systems such as HDFS, and the like, and the input source system and the output target system may be the same or different; the serialization can be realized by adopting a CSV format, and can also be realized by adopting JSON, XML, AVRO, PROTOBUF and other formats and custom formats. The embodiment of the present invention is not particularly limited thereto.

In the apply rules operator, the "anti-fraud" decision flow created above is specified as the rule in use. The rule in the application rule operator may be any type of rule, which is not specifically limited in this embodiment of the present invention.

When the rule operator is applied, the attributes (i.e., fields) of the event (each transaction record) are treated as rule variables of the rule, and the output result of the rule is treated as the attribute of the output event. In this case, the rule variables include "type", "transaction status", "amount", "transaction time", "first time consumption failure time", "number of consumption failures", and "amount consumed". The "first consumption failure time", "consumption failure times" and "consumption amount" are state variables, and after one event is processed, the value of the state variable remains unchanged and can be accessed during the next event.

For example, the input is three consecutive transaction records, as shown in Table 13 below:

serial number	Type (B)	Status of transaction	Amount of money	Transaction time
					1	Consumption of	Failure of	1000	2019.01.01 00:00:00
2	Consumption of	Failure of	800	2019.01.01 00:00:30
					3	Consumption of	Failure of	500	2019.01.01 00:01:00

TABLE 13 three consecutive transaction records

Before processing an event, initializing a variable 'consumption failure times' to be 0;

the first line entering the system is transaction record 1, and the record is processed: its "type" is "consume", "transaction status" is "fail", and the current "consumption failure times" is 0, thus satisfying rules 1, 2 and 3, setting and saving: the "consumption amount" is the "amount" of the transaction record 1 and the "number of consumption failures" is 1,

the "first time consumption failure time" is the "transaction time" of transaction record 1.

Then entering into the system as a transaction record 2, processing the record, and calculating and judging based on the saved data and the current data: the type is consumption, the transaction state is failure, the amount is 800, the transaction time is 2019.01.0100: 00:30, the current consumption failure times is 1, the consumption amount is 1000, the first consumption failure time is 2019.01.0100: 00:00, so that the rules 1 and 2 are met, the rule 3 is not met, the rule 4 is continuously judged to be met, and the following steps are set and stored: the "consumption amount" is the "amount" of the transaction record 2, and the "consumption failure times" is 2; and (3) continuing to judge that the rule 5 is not met, entering a system to be a transaction record 3, processing the record, and calculating and judging on the basis of the stored data and the current data: the type is consumption, the transaction state is failure, the amount is 500, the transaction time is 2019.01.0100:01:00, the current consumption failure times are 2, the consumption amount is 800, the first consumption failure time is 2019.01.0100: 00:00, so that the rules 1 and 2 are met, the rule 3 is not met, the rule 4 is continuously judged to be met, and the following steps are set and stored: the "consumption amount" is the "amount" of the transaction record 3, and the "consumption failure times" is 3; and continuously judging that the rule 5 is met, setting an output alarm mark, and setting and storing the consumption failure times to be 0.

As shown in fig. 6, the task of real-time streaming data is started, and the Kafka data input operator reads data from the designated input Kafka message queue in real time; the CSV deserialization operator analyzes the read data according to the format of the CSV and converts the read data into an event processed by a task of real-time stream data; processing the event by using an anti-fraud decision flow by using a rule engine in the rule operator; the CSV serialization operator encodes the processed event into data in a CSV format; the Kafka data output operator outputs the encoded data to a designated output Kafka message queue.

The above embodiments include the concepts of time sequence and state, where time sequence means that each piece of input data includes at least one timestamp attribute, and the data enters the system in an order that must be consistent with the order of the timestamps from small to large. In the above example, the time stamp takes the property of "transaction time" of the event, and the events must enter the application rule operator in the order of "transaction time" from small to large.

The events are ordered and the order is correct, namely the events are processed by order preservation, and the output result is not only related to the current input event but also related to the value of the state variable, and is indirectly related to the historical input event and the order thereof.

Based on the data real-time processing method provided in the above embodiment, an embodiment of the present invention further provides a data real-time processing system for implementing the above method, and referring to fig. 7, a data real-time processing system 700 provided in an embodiment of the present invention includes:

the human-computer interface display module 701 is used for displaying a user interface, receiving a first input, and generating an operator of a task of streaming data according to the first input; the operator comprises an applied rule operator comprising rules in at least one rule set

A task construction module 702, configured to construct a task of streaming data according to an operator of the task;

the running module 703 is configured to run the task of the stream data.

Optionally, the data real-time processing system 700 provided in the embodiment of the present invention further includes:

the rule interface module is used for displaying a rule creating interface;

wherein the set of rules contains at least one rule.

Optionally, the data real-time processing system further includes:

a selection module for selecting a target rule template from the list;

Optionally, the task further includes: operator and operator connection relation.

Optionally, the operation module 703 includes:

a receiving sub-module 7031 for receiving the stream data;

the calculation submodule 7032 is configured to calculate the stream data by using an application rule operator in the operators, and generate an output event;

and the output sub-module 7033 is used for outputting and displaying the output event.

Optionally, the application rule operator comprises a rule engine;

the calculation submodule 7032 is specifically configured to:

According to the data real-time processing system provided by the embodiment of the invention, the rule engine is placed in the operator of the data real-time processing system and is used for processing the streaming data in real time, so that the data processing efficiency and the real-time performance are greatly improved.

The embodiment of the invention provides a data real-time processing system, which comprises a processor, a memory and a computer program which is stored on the memory and can run on the processor, wherein when the computer program is executed by the processor, the steps of the data real-time processing method are realized.

The embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements each process of the data real-time processing method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. The computer-readable storage medium may be a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. A method for real-time processing of data, the method comprising:

displaying a user interface, receiving a first input, and generating an operator of a task of streaming data according to the first input; the operator comprises an application rule operator, the application rule operator comprises rules in at least one rule set, and the application rule operator comprises a rule engine;

constructing a task of streaming data according to the operator of the task, wherein the streaming data is data continuously generated by a data source;

running the task of the streaming data;

the step of running the task of the streaming data comprises:

receiving the streaming data;

outputting and displaying the output event;

2. The real-time data processing method according to claim 1, wherein the step of displaying the user interface and receiving a first input, and generating an operator of a task of streaming data based on the first input is preceded by the step of:

displaying a rule creation interface;

wherein the set of rules contains at least one rule.

3. The method of real-time processing of data according to claim 1, the method further comprising: based on the service scene of the application rule operator, extracting the same content in the multiple rules as a rule template, and storing the rule template in a template database.

4. The method of real-time processing of data according to claim 3, wherein the method further comprises:

inquiring the template database to obtain a list of the rule templates;

selecting a target rule template from the list;

and generating a corresponding rule according to the target rule template.

5. The real-time data processing method of claim 1, wherein the task further comprises: operator and operator connection relation.

6. The method of real-time processing of data according to claim 2, wherein the rules include at least one of: wizard rules, rule sets, score cards, decision tables, decision trees, decision flows, and models.

7. The method of real-time processing of data according to claim 6, wherein said decision flow is constituted by execution nodes organized in a directed graph structure, said execution nodes comprising: sub-rule node, routing node, start node.

8. The real-time data processing method according to claim 7, wherein the subsequent executing node of the routing node is determined according to a subsequent node calculation algorithm configured by the routing node, and the routing node is capable of ending the loop execution of the decision flow when a specified condition is satisfied.

9. The data real-time processing method of claim 1, wherein the rule variables include state variables whose values remain unchanged after one event process is completed and can be accessed at the time of the next event process.

10. The method of real-time processing of data of claim 6, wherein the model comprises a machine learning model, the machine learning model including an algorithm and associated parameters.

11. A system for real-time processing of data, comprising:

the human-computer interface display module is used for displaying a user interface, receiving first input and generating an operator of a task of streaming data according to the first input; the operator comprises an application rule operator, the application rule operator comprises rules in at least one rule set, and the application rule operator comprises a rule engine;

the task construction module is used for constructing a task of streaming data according to an operator of the task, wherein the streaming data is data continuously generated by a data source;

the running module is used for running the task of the streaming data;

the operation module comprises:

a receiving submodule for receiving the stream data;

the output submodule is used for outputting and displaying the output event;

the calculation submodule is specifically configured to:

12. The data real-time processing system of claim 11, wherein the data real-time processing system further comprises:

the rule interface module is used for displaying a rule creating interface;

wherein the set of rules contains at least one rule.

13. The data real-time processing system of claim 11, wherein the data real-time processing system further comprises:

14. The data real-time processing system of claim 13, wherein the data real-time processing system further comprises:

a selection module for selecting a target rule template from the list;

15. The data real-time processing system of claim 11, wherein the task further comprises: operator and operator connection relation.

16. The data real-time processing system of claim 12, wherein the rules include at least one of: wizard rules, rule sets, score cards, decision tables, decision trees, decision flows, and models.

17. The data real-time processing system of claim 16, wherein the decision flow is comprised of execution nodes organized in a directed graph structure, the execution nodes comprising: sub-rule node, routing node, start node.

18. The data real-time processing system of claim 17, wherein subsequent executing nodes of the routing node are determined according to a subsequent node computing algorithm configured by the routing node, the routing node being capable of ending the loop execution of the decision flow when a specified condition is satisfied.

19. The data real-time processing system of claim 11, wherein the rule variables include state variables whose values remain unchanged after one event process is completed and can be accessed at the next event process.

20. The data real-time processing system of claim 16, wherein the model comprises a machine learning model, the machine learning model including an algorithm and associated parameters.

21. A real-time processing system for data, comprising a processor, a memory and a computer program stored on the memory and executable on the processor, the computer program, when executed by the processor, implementing the steps of the real-time processing method for data according to any one of claims 1 to 10.

22. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method for real-time processing of data according to any one of claims 1 to 10.