CN111638883B

CN111638883B - Decision engine implementation method based on decision tree

Info

Publication number: CN111638883B
Application number: CN202010407619.XA
Authority: CN
Inventors: 谢世茂; 彭恒; 陈杰
Original assignee: Sichuan XW Bank Co Ltd
Current assignee: Sichuan XW Bank Co Ltd
Priority date: 2020-05-14
Filing date: 2020-05-14
Publication date: 2023-05-16
Anticipated expiration: 2040-05-14
Also published as: CN111638883A

Abstract

The invention relates to a decision engine implementation method based on a decision tree, which comprises the following steps: A. selecting needed decision nodes in a control dragging mode, and forming decision trees by the decision nodes; B. detecting the correctness of the decision tree; C. detecting the correctness of expressions contained in each node; D. analyzing the decision tree: analyzing all decision paths according to the branch structure of the decision tree, and converting information and configuration conditions contained in each node in each decision path into an SQL statement expression mode, so that each decision path is expressed by one SQL statement; E. and sending the SQL statement of the decision tree to a big data cluster for distributed computation and operation. The invention can generate an executable decision scheme without programming, greatly reduces the operation difficulty of a user, and enables the decision scheme with large data magnitude to be calculated and operated simultaneously in a distributed calculation mode.

Description

Decision engine implementation method based on decision tree

Technical Field

The invention relates to a data processing method in the financial field, in particular to a decision engine implementation method based on a decision tree.

Background

In the field of banks, along with the gradual expansion of financial scale, people summarize various rules about scenes such as wind control, anti-fraud, marketing and the like, and through the rules, users with higher overdue rate and lower credit can be identified, and users with better repayment capability and stronger loan willingness can also be identified. Through various rule types, decisions such as freezing, thawing, regulating the amount of money, interest rate and the like can be made for the user. The risk of bad assets of the bank can be reduced, and the income of the bank can be increased.

Current solutions for implementing banking financial policies are mainly implemented by purchasing third party rule engines, such as ilog, URule, etc. However, rule engines are generally expensive and rule configuration requires users to understand certain programming skills and are therefore generally more suitable for the pneumatic control sector, but are more difficult to learn for the operations or other business sectors. In addition, taking ilog as an example, the way that ilog provides services to the outside is to provide an API interface (application program interface), and this way has two main disadvantages:

a) Not suitable for mass business, as the data scale increases, the amount of data that banks may need to decide every day exceeds ten millions, at this time, if the ilog needs to be used continuously, several ilog servers need to be deployed to process the requests concurrently to meet timeliness, which undoubtedly further increases the resource consumption.

B) Different business departments need to develop different portals for invoking ilog or configuring ilog rules, greatly increasing the amount of tasks developed.

There is therefore a need for an implementation that does not require programming and that can be adapted to big data banking policies to accommodate the operation of many different types of departments, users.

Disclosure of Invention

The invention provides a decision engine implementation method based on a decision tree, which can generate an executable decision scheme without programming, reduces the operation difficulty of a user and can be suitable for decision calculation and operation of big data.

The invention discloses a decision engine implementation method based on a decision tree, which comprises the following steps:

A. selecting a needed decision node through a visual control dragging mode on a display device by using a signal input device, configuring the selected decision node, and establishing a father/son node relation according to a logic relation among the decision nodes so as to form a decision tree from a heel node to a leaf node;

B. detecting correctness of the decision tree, including detecting correctness of necessary field information in codes corresponding to the decision tree, correctness of father nodes and child nodes of each node and correctness of condition setting of each node;

C. detecting correctness of expressions contained in each node, including detecting correctness of expression formats and correctness of functions and variables contained in the expressions;

D. analyzing the decision tree: all decision paths are analyzed according to the branch structure of the decision tree by traversing the decision tree, and information and configured conditions contained in each node in each decision path are converted into SQL sentence expression modes, so that each decision path is expressed by one SQL sentence;

E. and sending the SQL statement of the decision tree to a big data cluster for distributed computation and operation.

The invention constructs the decision tree in a dragging mode by adopting the visualized page, and then configures rules for each decision node, so that the invention is very simple to use, does not need to carry out actual programming, and greatly reduces the use difficulty of users. The decision tree is analyzed into the scripts of a plurality of SQL sentences, and the scripts are submitted to a big data cluster for distributed computation, so that the computation efficiency of tens of millions of data decisions is ensured. In addition, the invention does not interact with the outside in an API (application program interface) mode, a user directly configures needed decisions on a decision tree (decision engine), and the problem that a business department needs to input manpower to develop calling programs is avoided. Because only the construction of the decision tree and the conversion of the SQL sentence are local, the execution and the calculation of the SQL sentence are not local, but distributed calculation is carried out through a large data cluster, so that the actual calculation and processing capacity is greatly improved, the data with large data magnitude can be calculated in parallel, the traditional decision engine scheme data are all in a local memory, and the calculation is carried out one by one, so that the concurrency effect is weaker. In the invention, the calculation tasks are distributed to each kernel corresponding to each node for concurrent calculation, and the larger the decision scale is, the faster the decision speed is compared with the traditional scheme.

Specifically, the step B includes:

B1. analyzing json format codes corresponding to the decision tree, wherein each object contained in the json format codes represents a corresponding node, each node contains a node ID and an associated node ID, and if the json format codes cannot analyze or lack necessary fields, the decision tree analysis exception error is thrown out;

B2. according to the information of each node in the decision tree, establishing an adjacency list for saving the tree structure of the decision tree in a memory, detecting the number of father nodes of each node, and throwing out an incidence abnormality if the number of father nodes of a certain node is more than 1;

B3. judging whether a loop or a circuit break exists in the decision tree through a parallel checking algorithm, and correspondingly throwing out the loop abnormality or the forest abnormality if the loop or the circuit break exists;

B4. detecting whether information loss exists in the condition setting of each node, if so, throwing out information loss abnormality, and displaying node names of specific missing information.

Specifically, the step C includes:

C1. scanning expressions contained in each node, and detecting whether abnormal characters exist or not;

C2. extracting functions and variables contained in all expressions through regular expressions, sequentially judging whether the variables exist in a basic field table or a declared function table, and throwing out errors of unknown variables or unknown functions of the expressions if the variables do not exist;

C3. and carrying out grammar detection on the expression, and throwing out the grammar error exception of the expression if grammar error exists.

Further, in step C1, if the contents in the quotation marks in the expression are removed, other characters remain in the expression, and it is determined that the expression is illegal.

Specifically, step D includes:

D1. obtaining a graph structure in the form of an adjacency list of a decision tree;

D2. traversing the whole decision tree from the root node of the decision tree through a depth-first search algorithm according to the graph structure, and recording the father node of each node in the traversing process;

D3. and D2, traversing all leaf nodes in a circulating way, recursively backtracking to a root node according to all father nodes of each leaf node recorded in the step D2, obtaining a condition list of SQL sentences according to the conditions configured in each node in the backtracking process, and forming the SQL sentence expression of each decision path after assembling the contents in the condition list.

According to the decision engine implementation method based on the decision tree, an executable decision scheme can be generated without programming, so that the operation difficulty of a user is greatly reduced, and the decision scheme with large data magnitude can be calculated and operated simultaneously in a distributed calculation mode. It took about only 1-2 minutes to test a cluster of 6 nodes to perform about 300 ten thousand decisions.

The foregoing of the invention will be described in further detail with reference to the following detailed description of the examples. It should not be understood that the scope of the above subject matter of the present invention is limited to the following examples only. Various substitutions and alterations are also possible, without departing from the spirit of the invention, and are intended to be within the scope of the invention.

Drawings

FIG. 1 is a flow chart of a decision engine implementation method based on decision trees of the present invention.

Fig. 2 is a schematic diagram of the structure of the adjacency list of the sample a in the step B2.

Fig. 3 is a schematic diagram of the structure of the adjacency list of the sample B in the step B2.

Fig. 4 is a schematic diagram of the structure of the adjacency list of the sample a in the step B3.

Fig. 5 is a schematic diagram of the structure of the adjacency list of the sample B in the step B3.

Fig. 6 is a schematic diagram of the decision tree structure obtained in step D1.

FIG. 7 is a schematic diagram of a decision path obtained from the decision tree of FIG. 6.

Detailed Description

The decision engine implementation method based on the decision tree of the invention as shown in fig. 1 comprises the following steps:

A. and selecting a needed decision node through a visual control dragging mode on a display device by using a signal input device, configuring the selected decision node, and establishing a father/son node relation according to a logic relation among the decision nodes so as to form a decision tree from a heel node to a leaf node.

The system for constructing the decision tree by dragging the decision node is an existing system, and the system is not described in detail herein, and the implementation principle can refer to various existing software with a dragging control function, such as Visio, dreamweaver and the like.

B. Detecting correctness of the decision tree, including:

B1. analyzing json format codes corresponding to the decision tree, wherein each object contained in the json format codes represents a corresponding node, each node contains a node ID and an associated node ID, and if the json format codes cannot analyze or lose necessary fields, the decision tree analysis exception error is thrown out. For example:

sample a: there are json as follows:

[{"node_id":"a","next_nodes":["b","c"]},{"node_id":"b","next_nodes":[]},{"node_id":"c","

next_nodes":[]}]

the json field in the actual code is richer and is only exemplified here. The sample is in a normal json format, and json resolution can be accomplished, so that detection will be passed.

In this and subsequent examples of this embodiment, node_id is used to record the current node number and next_nodes is used to record the next node to that node.

Sample b: there is json as follows:

[{"node_id":"a"}]

when the node is not a leaf node, a child node must be included, and at this time, the next_nodes information cannot be resolved (necessary key is missing), and the sample throws out the resolving exception.

B2. According to the information of each node in the decision tree, an adjacency list for saving the tree structure of the decision tree is built in a memory, the number of father nodes of each node is detected, and if the number of father nodes of a certain node is more than 1, the exception of the degree of entry is thrown out.

Sample a: constructing an adjacency list according to the analysis result of the last step, and assuming that the obtained adjacency list is in the following relation: a- > b, b- > c, b- > d, b- > e, are plotted in a visual form, and the graph structure is shown in fig. 2.

It can be seen that the structure is a standard tree structure and that no nodes in the structure have an ingress of greater than 1. The passage can then be detected at this point.

Sample b: assume that the adjacency list relationship is: a- > b, a- > c, b- > d, c- > d, are plotted in a visual form, and the graph structure is shown in fig. 3.

It is apparent that the node d of fig. 3 has an entry of 2, which does not conform to the definition of the tree in the data structure, and an exception of the entry is thrown.

B3. And judging whether a loop or a circuit break exists in the decision tree through a union checking algorithm, and correspondingly throwing out the loop abnormality or the forest abnormality if the loop or the circuit break exists. For example:

sample a: assume that the adjacency list relationship is: the visualization of a- > b, b- > c, c- > a is shown in fig. 4.

In this case, although the ingress degree of each node is 1, a loop exists in the structure, and the situation is not in accordance with the definition of the tree in the data structure, the loop in the graph structure can be identified through a union algorithm, and the user is notified of the loop abnormality when the loop is found.

Sample b: assume that the adjacency list relationship is: a- > b, c- > d, as shown in fig. 5.

At this time, the situation that no loop exists in the graph structure and the ingress degree is larger than 1 does not exist, but at this time, the graph can be found to be not communicated, at this time, forest anomalies can be found through a union algorithm, and a user is notified.

B4. Detecting whether information loss exists in the condition setting of each node, if so, throwing out information loss abnormality, and displaying node names of specific missing information. For example:

sample a: suppose that the json structure of a node is received as follows:

{"expr":"max((crdt_limit/aval_limit)*crdt_limit,300000)","operator_type":">","value":""}

the information of the graph structure is hidden in the json structure, and only the data related to the node condition is reserved. It can be found that in the json structure, the value is not filled in to be null, and the user is notified that the information is missing. Meanwhile, whether the expression of the field of the expr is abnormal or not is checked through the detection of the expression, and if no abnormality exists, the decision tree can be normally analyzed into an SQL sentence.

In this embodiment, the expr represents the content of the expression, the operator_type represents an operator, including '>', '<','=', '<=', '=', five symbols, and the value represents a specific comparison object, where a constant may be filled in.

C. Detecting correctness of expressions contained in each node includes:

C1. and scanning expressions contained in each node, and detecting whether abnormal characters exist. The character string is allowed to have only numerical values, letters, underlines, brackets, additions, subtractions, multiplications, divisors (modular operations), commas, decimal points, quotients. If other characters remain after the content in the quotation marks is removed, the expression is directly judged to be illegal. For example:

sample a: max ((crdt_limit/aval_limit) ×crdt_limit, 300000), this time will be verified successfully, because all symbols in the expression are legal characters.

In the expression, max is a maximum function taking two values, aval_limit is a usable credit limit, crdt_limit is a credit limit (maximum credit limit) of the user, and the meaning of the expression is that the credit of the user is improved according to the credit use condition of the user, but the maximum credit is increased to 30 ten thousand.

Sample b: crdt_limit ζ.2, at which time the anomaly symbol is scanned out, thereby throwing an error.

C2. Extracting functions and variables contained in all the expressions through the regular expressions '[ A-Za-z ] [ A-Za-z_0-9] +' and sequentially judging whether the variables exist in a basic field table or a declared function table, and if not, throwing out that the expressions have unknown variables or errors of the unknown functions. For example:

sample a: max ((crdt_limit/aval_limit) ×crdt_limit, 300000), at this time, the 4 strings max, aval_limit, crdt_limit are read out by the expression, and by comparing function tables and field tables of the 4 strings in the database, whether there is an undefined function or variable is queried, and if all are defined, verification is passed.

Sample b: the maxx (sample 1+sample2, 20) extracts 3 strings of the maxx, sample1 and sample2, and by comparison, the maxx is neither a variable nor a function, and throws out undefined exceptions.

C3. And carrying out grammar detection on the expression, and throwing out the grammar error exception of the expression if grammar error exists. The first method is to directly carry out recursion analysis on the expression by using a recursion descent analysis method to judge whether the expression passes grammar detection or not. The second is to compile the expression into a script, and detect it with a script compiler. In the second embodiment, the compiling of the expression into the grovy script completes the grammar detection. If no expression anomaly is found, it is indicated that the expression is correct. For example:

sample a: max ((crdt_limit/aval_limit) ×crdt_limit, 300000), the expression will pass verification.

Sample b: max (aatt, bbcc), the expression lacks significantly right brackets, at which point syntax errors are found.

D. Analyzing the decision tree:

D1. a graph structure in the form of an adjacency list of the decision tree is obtained. For example:

sample example: it is assumed that a decision tree as shown in fig. 6 is obtained. Fig. 6 is a standard binary tree structure, and each node is filled with the correct expression condition and other information as required, so that the detection passes.

The credit field is named crdt_limit, the field that has been overdue is named is_overdie, and the field that is blacklisted is named is_black. When resolving the decision tree, all the related field names are resolved into the expression mode of SQL sentences, and the fields used in the SQL sentences are English of the three names.

D2. According to the graph structure, traversing the whole decision tree from the root node of the decision tree through a depth-first search algorithm, and recording the father node of each node in the traversing process. Taking the decision tree of fig. 6 as an example, with 4 leaf nodes in fig. 6, we can calculate 4 decision paths using a depth-first algorithm, as shown in fig. 7, respectively.

D3. And D2, traversing all leaf nodes in a circulating way, recursively backtracking to a root node according to all father nodes of each leaf node recorded in the step D2, obtaining a condition list of SQL sentences according to the conditions configured in each node in the backtracking process, and forming the SQL sentence expression of each decision path after assembling the contents in the condition list. Still taking fig. 6 and 7 as examples:

after parsing the 4 decision paths shown in FIG. 7, each path is converted to SQL conditions. The SQL conditions for these 4 decision paths are:

A)crdt_limit>20000,is_overdue＝1；

B)crdt_limit>20000,is_overdue＝0；

C)crdt_limit<＝20000,is_black＝1；

D)crdt_limit<＝20000,is_black＝0；

and then the obtained condition list is assembled into SQL sentences, and each leaf node corresponds to one SQL sentence.

Splicing the SQL conditions to obtain 4 SQL sentences, wherein the SQL sentences are respectively as follows:

A)select(crdt_limit/2)from search_table where(1＝1)and(crdt_limit>20000)and(is_overdue＝1)；

B)select(crdt_limit*1.2)from search_table where(1＝1)and(crdt_limit>20000)and(is_overdue＝0)；

c) select ('freeze') from search_table sphere (1=1) and (crdt_limit < =20000) and (is_black=1);

d) select ('no action') from search_table sphere (1=1) and (crdt_limit < =20000) and (is_black=0);

It took about only 1-2 minutes to test a cluster of 6 nodes to perform about 300 ten thousand decisions.

The core idea of the invention is that the method is similar to the operation mode of spark (a fast and universal computing engine for large-scale data processing), only tasks are analyzed and submitted locally, and the tasks are computed by using a large data cluster. In large data clusters, data is stored in a distributed manner in each node, and the data quantity which can be calculated simultaneously also far exceeds the memory size of a local single node. The present invention is able to complete millions of decisions at once. While the traditional decision engine scheme relies on local memory for calculation, if the local memory is too small, it is difficult to support large-scale calculation. If the support is needed, the data is needed to be segmented and calculated in batches, so that the traditional scheme is far more complicated than the proposal in programming realization and data management if the effect of the same scale is to be achieved.

In addition, the invention has very high realization speed for batch decision. The traditional decision engine scheme data are in a local memory, and calculation is required to be carried out piece by piece during calculation, so that the concurrency effect is weaker. Most of the computation is not local, the computation task can be distributed to each kernel of each node to perform concurrent computation, and the larger the decision scale, the faster the decision speed is compared with the traditional scheme.

Claims

1. The decision engine implementation method based on the decision tree is characterized by comprising the following steps:

E. the SQL sentence of the decision tree is sent to a big data cluster to perform distributed computation and operation;

the step B comprises the following steps:

2. The decision engine implementation method based on decision tree as claimed in claim 1, characterized in that: the step C comprises the following steps:

3. The decision engine implementation method based on decision tree as claimed in claim 2, characterized in that: in step C1, if the contents in the quotation marks in the expression are removed, other characters still remain in the expression, and it is determined that the expression is illegal.

4. The decision engine implementation method based on decision tree as claimed in claim 1, characterized in that: the step D comprises the following steps: