CN111198863A

CN111198863A - Rule engine and implementation method thereof

Info

Publication number: CN111198863A
Application number: CN201911378555.9A
Authority: CN
Inventors: 许聪慧; 刘平怀; 陈涛
Original assignee: Tianyang Hongye Technology Co ltd
Current assignee: Tianyang Hongye Technology Co ltd
Priority date: 2019-12-27
Filing date: 2019-12-27
Publication date: 2020-05-26
Anticipated expiration: 2039-12-27
Also published as: CN111198863B

Abstract

The embodiment of the application discloses a rule engine and an implementation method thereof, wherein a rule configuration request message sent by a rule engine page is received, and the rule configuration request message carries configuration rule information; compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script; sending the SPARK cluster execution script to a distributed file system hdfs; submitting one or more task information to the SPARK cluster; and receiving one or more calculation results sent by the SPARK cluster, wherein the calculation results are obtained by reading corresponding data information in the hdfs or the database by the SPARK cluster according to the task information and calculating. The capability of the rule engine for calculating the rule by using the memory is improved.

Description

Rule engine and implementation method thereof

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to a rule engine and an implementation method thereof.

Background

The implementation method of the batch operation of the rule engines in the big data environment has the following mechanism probably: firstly, a drools rule is generated through a rule engine, and the rule is deployed through a planning engine platform. The traditional server cluster has the following calculation mode: and acquiring a large amount of data of the database according to the transmitted rule number, performing rule calculation in a multithreading mode, and finally analyzing and warehousing the calculation result. The big data platform has the following calculation mode: and executing the rule execution package through the big data platform to perform rule calculation according to the rule number and the path of the incoming data file, and storing the calculated result into a big data file.

In the prior art, a rule calculation engine runs batch through a server cluster and excessively depends on server resources, so that the server resources cannot be reused; moreover, in the process of rule calculation for mass data on a big data platform, the performance bottleneck is easy to reach, and the user requirement cannot be met. For executing the rules on the big data platform, the bottom layer still uses drools to perform rule calculation, so that the calculation capability of the big data platform cannot be fully utilized.

Disclosure of Invention

Therefore, the embodiment of the application provides a rule engine and an implementation method thereof, and the capability of utilizing a memory to perform rule calculation is improved.

In order to achieve the above object, the embodiments of the present application provide the following technical solutions:

according to a first aspect of an embodiment of the present application, there is provided a method for implementing a rule engine, the method including:

receiving a rule configuration request message sent by a rule engine page, wherein the rule configuration request message carries configuration rule information;

compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script;

sending the SPARK cluster execution script to a distributed file system hdfs;

submitting one or more task information to the SPARK cluster;

and receiving one or more calculation results sent by the SPARK cluster, wherein the calculation results are obtained by calculating according to data information read by the SPARK cluster in the hdfs or database according to the task information.

Optionally, the SPARK cluster execution script is a JAR file; the compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script comprises the following steps: and analyzing the configuration rule information into java codes, and compiling and packaging the java codes into JAR files.

Optionally, after the receiving one or more computation results sent by the SPARK cluster, the method further includes: and sending the one or more rule calculation results to the hdfs for storage.

Optionally, the database information includes a parameter mapping relationship, data model information and calculation data, where the data model information includes a variable definition, and the calculation data includes configuration rule information; the task information comprises a data file corresponding to the submitted task or database connection and SQL information.

Optionally, the calculation result is obtained by configuring an hdfs path or database connection and SQL information by the SPARK cluster according to the task information, and reading corresponding data information in the hdfs or the database for calculation.

According to a second aspect of embodiments of the present application, there is provided a rules engine comprising:

the message receiving module is used for receiving a rule configuration request message sent by a rule engine page, wherein the rule configuration request message carries configuration rule information;

the compiling module is used for compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script;

the first sending module is used for sending the SPARK cluster execution script to a distributed file system hdfs;

the second sending module is used for submitting one or more task information to the SPARK cluster;

and the result receiving module is used for receiving one or more calculation results sent by the SPARK cluster, wherein the calculation results are obtained by calculating the SPARK cluster by reading corresponding data information in the hdfs or a database according to the task information.

Optionally, the SPARK cluster execution script is a JAR file; the compiling module is specifically configured to parse the configuration rule information into java codes, and compile and package the java codes into JAR files.

Optionally, the first sending module is further configured to: and sending the one or more rule calculation results to the hdfs for storage.

Optionally, the calculation result is obtained by configuring an hdfs path or database connection and SQL information by the SPARK cluster according to the task information, and reading corresponding data information in the hdfs or database for calculation.

To sum up, in the rule engine and the implementation method thereof provided in the embodiment of the present application, a rule configuration request message sent by a rule engine page is received, where the rule configuration request message carries configuration rule information; compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script; sending the SPARK cluster execution script to a distributed file system hdfs; submitting one or more task information to the SPARK cluster; and receiving one or more calculation results sent by the SPARK cluster, wherein the calculation results are obtained by calculating according to data information read by the SPARK cluster in the hdfs or database according to the task information. The capability of utilizing the memory to calculate the rule is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It should be apparent that the drawings in the following description are merely exemplary, and that other embodiments can be derived from the drawings provided by those of ordinary skill in the art without inventive effort.

The structures, ratios, sizes, and the like shown in the present specification are only used for matching with the contents disclosed in the specification, so that those skilled in the art can understand and read the present invention, and do not limit the conditions for implementing the present invention, so that the present invention has no technical significance, and any structural modifications, changes in the ratio relationship, or adjustments of the sizes, without affecting the functions and purposes of the present invention, should still fall within the scope of the present invention.

Fig. 1 is a method for implementing a rule engine according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an embodiment of a rules engine method according to the present application;

FIG. 3 is a schematic diagram of a rule engine method according to another embodiment of the present application;

fig. 4 is a block diagram of a rule engine according to an embodiment of the present application.

Detailed Description

The present invention is described in terms of particular embodiments, other advantages and features of the invention will become apparent to those skilled in the art from the following disclosure, and it is to be understood that the described embodiments are merely exemplary of the invention and that it is not intended to limit the invention to the particular embodiments disclosed. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

SPARK is a fast, general-purpose computing engine designed specifically for large-scale data processing. The rule engine is developed by an inference engine, is a component embedded in an application program, and realizes the separation of business decisions from application program codes and the writing of the business decisions by using a predefined semantic module. And receiving data input, interpreting business rules, and making business decisions according to the business rules. Most rule engines support rule order and rule conflict checking, support rule implementation of simple scripting languages, and support embedded development of universal development languages. There are a number of rules engines currently available in the industry, including business and open source selection.

A Distributed File System (HDFS) refers to a Distributed File System designed to fit on general purpose hardware. It has many similarities with existing distributed file systems. But at the same time, its distinction from other distributed file systems is also clear. HDFS is a highly fault tolerant system suitable for deployment on inexpensive machines. HDFS provides high throughput data access and is well suited for application on large-scale data sets.

Fig. 1 shows an implementation method of a rule engine provided in the embodiment of the present application, which improves the capability of performing rule calculation by using a memory. A dependency item is added on the basis of the existing executor, and the functions of the original executor are enhanced, so that SPARK task submission, rule compilation and interaction with a rule engine are supported through an interface. And organizing, compiling and transforming components such as a decision tree, a decision table, a scoring model, a code block, a rule flow, data mining, a rule set and the like in the rule engine platform to support the SPARK platform to execute tasks. The implementation method of the rule engine comprises the following steps:

step 101: and receiving a rule configuration request message sent by a rule engine page, wherein the rule configuration request message carries configuration rule information.

Step 102: and compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script.

Step 103: and sending the SPARK cluster execution script to a distributed file system hdfs.

Step 104: submitting one or more task information to the SPARK cluster.

Step 105: and receiving one or more calculation results sent by the SPARK cluster, wherein the calculation results are obtained by calculating according to data information read by the SPARK cluster in the hdfs or database according to the task information.

In one possible embodiment, the SPARK cluster execution script is a JAR file; in step 102, the configuration rule information is parsed into java codes, and the java codes are compiled and packaged into JAR files. The JAR file is cross-platform, so no concern is given about the problems involved with a particular platform. In addition, each file is uncompressed. But after all these files are merged into one JAR file, only one request needs to be sent to the remote server. Due to the compression technique, the entire data can be acquired in a shorter time. In short, some classes written in the JAR file are packaged and introduced into the project, and the classes, the attributes and the method in the JAR file are directly used.

In one possible embodiment, in step 104, the rule script is packaged and the SPARK cluster is submitted remotely for batch task execution. The task can be submitted in two ways, namely: exporting a rule JAR file, manually uploading the rule JAR file to the SPARK cluster, and initiating a task by using a command line; the second method comprises the following steps: the tasks are submitted to the SPARK cluster by submitting the tasks using the SPARK launcher API provided by SPARK.

In one possible embodiment, FIG. 2 illustrates the establishment of a mapping between variables and data required to execute a rule script in a configured form by associating input data with rule parameters. In step 105, the data reading supports two modes, one mode is that the existing data in the database is read through SPARK SQL aiming at the database connection information and the SQL content; the other is for the hdfs data file path, which reads data through a file, such as reading data on a hdfs big data platform.

In one possible embodiment, the database information includes parameter mapping relationships, data model information including, but not limited to, variable definitions, and calculation data including, but not limited to, configuration rule information; the task information comprises a data file corresponding to the submitted task or database connection and SQL information.

In a possible implementation manner, the calculation result is obtained by configuring an hdfs path or database connection and SQL information by the SPARK cluster according to the task information, and reading corresponding data information in the hdfs or database for calculation. Java Database Connectivity (JDBC) is an application program interface in the Java language that specifies how client programs access databases, providing methods such as querying and updating data in databases.

In one possible implementation, after step 105, the one or more rule calculation results are sent to the hdfs for storage. And the storage mode of the configuration execution result is supported, such as the execution result is stored into a json file, a big data file or directly put into a warehouse for storage.

In summary, it can be seen that the operations of rule configuration, test, saving, and the like of the rule engine provided in the embodiment of the present application are not changed, and only one "button" is added for viewing and checking java code applied to SPARK. And the advantages of SPARK memory calculation can be fully utilized due to the drools dependence separated in the execution process, so that the dependence on the drools is reduced, and the code is autonomously controllable. Under the big data environment, the computing power of the rule engine is greatly improved.

In order to make the implementation method of the rule engine provided in the embodiment of the present application clearer, an example of the flow is now described with reference to fig. 3:

step 1: and (4) configuring rules on the rule engine page, and configuring the rules on the rule engine page popped up by the system. The calculated data supports json format (including input and output data) by default, custom configuration is added, the columnar data file can be read by configuring a data mapping relation and a data segmentation symbol, and the output data is the same as the data. Besides configuring specific rules, the rule engine also needs to configure corresponding parameters, such as data file paths, SQL information, data mapping relationships, and the like, which are necessary information for tasks, during online deployment.

Step 2: and the rule engine executor receives the instruction sent by the rule engine page, compiles the rule and packs the rule into an executable JAR package. Namely, the rule engine executor reads the information of the rule engine, executes java codes of the compiling rules, and prints the java codes into an executable rule JAR package which is uploaded to the hdfs of the distributed file system. I.e., converting the rules on the rules engine platform into a SPARK executable script. The system supports SPARK local operation, and can execute the regular JAR packet by a single machine as long as a SPARK environment exists locally. In addition, the rule JAR packet can be exported and directly uploaded to any SPARK cluster platform to be operated.

And 3, step 3: the executable rule JAR package is submitted to hdfs.

And 4, step 4: and submitting task information to a rule engine SPARK cluster. Support mode 1: and exporting a rule JAR packet, manually uploading the rule JAR packet to the SPARK cluster, and initiating a task by using a command line. Support mode 2: the tasks are submitted to the SPARK cluster by submitting the tasks using the SPARK launcher API provided by SPARK.

And 5, step 5: the rule engine SPARK cluster reads database information, the read data is determined by a data file or sql specified by a submission task, and the database information comprises mapping relation, data model (information such as variable definition), calculation data and the like (the read information is related to specific rules). After data is read, firstly, the data is analyzed, and the data is converted into corresponding input parameters according to the mapping relation, and the input parameters can be directly applied to the rules for calculation. The rule JAR package supports direct reading of data information from a hadoop platform and a database (Mysql, Oracle), and loading of the data information into a SPARK cluster (the hdfs path needs to be configured, or the database connection and sql information need to be configured).

And 6, step 6: and the rule engine SPARK cluster returns the calculation result to the rule engine system so that the rule engine system executes the rule of the rule JAR packet, calculates the data and stores the result into hdfs.

In summary, the embodiment of the present application provides a method for implementing a rule engine, where a rule configuration request message sent by a rule engine page is received, where the rule configuration request message carries configuration rule information; compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script; sending the SPARK cluster execution script to a distributed file system hdfs; submitting one or more task information to the SPARK cluster; and receiving one or more calculation results sent by the SPARK cluster, wherein the calculation results are obtained by calculating according to data information read by the SPARK cluster in the hdfs or database according to the task information. The capability of utilizing the memory to calculate the rule is improved. Only one new task for deploying batches and a function for initiating the tasks are added, and the change of the original rule engine is extremely small.

Based on the same technical concept, an embodiment of the present application provides a rule engine, as shown in fig. 4, the rule engine includes:

the message receiving module 401 is configured to receive a rule configuration request message sent by a rule engine page, where the rule configuration request message carries configuration rule information.

And the compiling module 402 is configured to compile the configuration rule information and convert the configuration rule information into an SPARK cluster execution script.

A first sending module 403, configured to send the SPARK cluster execution script to the distributed file system hdfs.

A second sending module 404, configured to submit one or more task information to the SPARK cluster.

A result receiving module 405, configured to receive one or more calculation results sent by the SPARK cluster, where the calculation results are obtained by calculating, according to the task information, data information corresponding to the SPARK cluster read in the hdfs or a database.

In one possible embodiment, the SPARK cluster execution script is a JAR file; the compiling module 402 is specifically configured to parse the configuration rule information into java codes, and compile and package the java codes into JAR files.

In a possible implementation manner, the first sending module 403 is further configured to: and sending the one or more rule calculation results to the hdfs for storage.

In one possible embodiment, the database information includes parameter mapping relationships, data model information, and calculation data, wherein the data model information includes variable definitions, and the calculation data includes configuration rule information; the task information comprises a data file corresponding to the submitted task or database connection and SQL information.

In a possible implementation manner, the calculation result is that the SPARK cluster configures an hdfs path or database connection or SQL information according to the task information, and corresponding data information is read from the hdfs or database and calculated.

In the present specification, each embodiment of the method is described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. Reference is made to the description of the method embodiments.

It is noted that while the operations of the methods of the present invention are depicted in the drawings in a particular order, this is not a requirement or suggestion that the operations must be performed in this particular order or that all of the illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.

Although the present application provides method steps as in embodiments or flowcharts, additional or fewer steps may be included based on conventional or non-inventive approaches. The order of steps recited in the embodiments is merely one manner of performing the steps in a multitude of orders and does not represent the only order of execution. When an apparatus or client product in practice executes, it may execute sequentially or in parallel (e.g., in a parallel processor or multithreaded processing environment, or even in a distributed data processing environment) according to the embodiments or methods shown in the figures. The terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, article, or apparatus that comprises the recited elements is not excluded.

The units, devices, modules, etc. set forth in the above embodiments may be implemented by a computer chip or an entity, or by a product with certain functions. For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. Of course, in implementing the present application, the functions of each module may be implemented in one or more software and/or hardware, or a module implementing the same function may be implemented by a combination of a plurality of sub-modules or sub-units, and the like. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may therefore be considered as a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.

The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, or the like, and includes several instructions for enabling a computer device (which may be a personal computer, a mobile terminal, a server, or a network device) to execute the method according to the embodiments or some parts of the embodiments of the present application.

The embodiments in the present specification are described in a progressive manner, and the same or similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. The application is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

The above-mentioned embodiments are further described in detail for the purpose of illustrating the invention, and it should be understood that the above-mentioned embodiments are only illustrative of the present invention and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.

Claims

1. A method for implementing a rules engine, the method comprising:

sending the SPARK cluster execution script to a distributed file system hdfs;

submitting one or more task information to the SPARK cluster;

2. The method of claim 1, wherein the SPARK cluster execution script is a JAR file;

the compiling the configuration rule information and converting the configuration rule information into an SPARK cluster execution script comprises the following steps:

and analyzing the configuration rule information into java codes, and compiling and packaging the java codes into JAR files.

3. The method of claim 1, wherein after the receiving one or more computation results sent by the SPARK cluster, the method further comprises:

and sending the one or more rule calculation results to the hdfs for storage.

4. The method of claim 1, wherein the database information includes parameter mapping relationships, data model information, and calculation data, wherein the data model information includes variable definitions, and the calculation data includes configuration rule information;

the task information comprises a data file corresponding to the submitted task or database connection and SQL information.

5. The method of claim 1, wherein the calculation result is obtained by configuring an hdfs path or database connection and SQL information by the SPARK cluster according to the task information, and reading corresponding data information in the hdfs or database for calculation.

6. A rules engine, wherein the rules engine comprises:

7. The rules engine of claim 6, wherein the SPARK cluster execution script is a JAR file;

the compiling module is specifically configured to parse the configuration rule information into java codes, and compile and package the java codes into JAR files.

8. The rules engine of claim 6, wherein the first sending module is further to:

and sending the one or more rule calculation results to the hdfs for storage.

9. The rules engine of claim 6, wherein the database information comprises parameter mapping relationships, data model information, and calculation data, wherein the data model information comprises variable definitions, and the calculation data comprises configuration rule information;

10. The rules engine of claim 6, wherein the computation result is obtained by the SPARK cluster configuring hdfs paths or database connections and SQL information according to the task information, and reading corresponding data information in hdfs or a database for computation.