CN109710233A - A kind of index operation method of business risk regulation engine - Google Patents
A kind of index operation method of business risk regulation engine Download PDFInfo
- Publication number
- CN109710233A CN109710233A CN201811604910.5A CN201811604910A CN109710233A CN 109710233 A CN109710233 A CN 109710233A CN 201811604910 A CN201811604910 A CN 201811604910A CN 109710233 A CN109710233 A CN 109710233A
- Authority
- CN
- China
- Prior art keywords
- server
- servers
- index
- regulation engine
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
A kind of index operation method of business risk regulation engine, comprising the following steps: Spark cluster S1, is configured on director server；S2, the setting target script drive module on director server are arranged logic control parameter when script drive module is arranged and SparkContext, the logic control parameter are transferred to the Cluster manager of Spark cluster by SparkContext；Index processor active task is assigned to the Cluster manager in Spark cluster by S3, script drive module；S4, pass through MapReduce mechanism, whole index processor active task is disassembled；" busy extent " of S5, Cluster manager by the index processor active task after dismantling according to other servers is mounted on other relatively idle servers；Implementing result is transmitted to cache module and is stored and be back to director server by S6, every server after having executed index processor active task.The present invention can quickly be calculated for the regulation engine of semi-structured text as a result, alleviating the situation of regulation engine computing capability deficiency.
Description
Technical field
The present invention relates to computer science software information technical fields, more particularly to a kind of business risk regulation engine
Index operation method.
Background technique
Regulation engine be widely used in recent years finance and it is counter cheat field, help monitors and finds target customers
In exception, risk, business opportunity etc..Most regulation engine can substantially be divided into two bulks in whole design, and one is rule
The building of system, secondly being the operation system construction of data flow.Currently, in the industry for the data used by regulation engine,
It mainly contains user behavior and (such as logs in, registers, browsing, collection, consumption) data, enterprise's financial data etc.；This kind of data
There are structuring, mensurable characteristic mostly.Such as user behavior data just be unable to do without number, frequency, price, time etc. generally
It reads.And in the regulation engine based on semi-structured text, index is the quantization to describe the certain concrete application scenes of client
Value, identical with number, the concept of frequency in the regulation engine of structuring, rule is a kind of logical comparison of index and threshold value.?
In entire regulation engine operation, for index as the bottom, the highest infrastructure elements of reusability, the efficiency of operation is direct
Affect the real-time of system.And when executing in batches rules results calculating, there are high concurrent, high connection number and whole calculation power
Insufficient problem.
Summary of the invention
In view of the above shortcomings of the prior art, the present invention provides a kind of index operation sides of business risk regulation engine
Method can be calculated quickly for the regulation engine of semi-structured text as a result, alleviating the insufficient feelings of regulation engine computing capability
Condition.
In order to solve the above-mentioned technical problem, present invention employs the following technical solutions:
A kind of index operation method of business risk regulation engine, comprising the following steps:
S1, Spark cluster is configured on director server, the IP address of the Servers-all in addition to director server is arranged
Into the Spark cluster of director server；
Logic control is arranged when script drive module is arranged in S2, the setting target script drive module on director server
Parameter and SparkContext, the logic control parameter are managed by the Cluster that SparkContext is transferred to Spark cluster
Manage device；
Index processor active task is assigned to the Cluster manager in Spark cluster by S3, script drive module；
S4, pass through MapReduce mechanism, the Cluster manager disassembles whole index processor active task；
" busy extent " of S5, Cluster manager by the index processor active task after dismantling according to other servers, carry
Onto other relatively idle servers；
Implementing result is transmitted to cache module and stored by S6, every server after having executed index processor active task
And it is back to director server.
As optimization, in step S5, judge that " busy extent " of other servers is sentenced according to Nginx load balancing
Disconnected.
As optimization, the Nginx realizes that the strategy of load balancing is poll distribution method, and each index processor active task is on time
Between sequence be assigned to other servers one by one, if a certain server is broken down, automatic rejection, remaining continuation poll.
As optimization, the Nginx realizes that the strategy of load balancing is Method for Weight Distribution, by monitoring other servers
The occupancy of CPU carrys out the weight of configuration access server, specifies the probability of access server, the weight and access probability are at just
Than.
As optimization, the cache module is cache, i.e. cache memory.
As optimization, the Cluster manager judges the nginx load balancing of " busy extent " of other servers
Strategy is poll distribution method, and each index processor active task is assigned to other servers one by one in chronological order, if a certain service
Device is broken down, automatic rejection, remaining continuation poll.
As optimization, the Cluster manager judges the nginx load balancing of " busy extent " of other servers
Strategy is Method for Weight Distribution, by monitor other servers CPU occupancy come the weight of configuration access server, specify and visit
Ask the probability of server, the weight is directly proportional with access probability.
The beneficial effects of the present invention are:
The present invention can quickly be calculated for the regulation engine of semi-structured text as a result, alleviating regulation engine meter
The case where calculating scarce capacity.
Detailed description of the invention
Fig. 1 is a kind of method flow diagram of the index operation method of business risk regulation engine of the present invention.
Fig. 2 is the local system architecture diagram of the cluster operation of index.
Specific embodiment
The present invention is described in further detail with reference to the accompanying drawing.
As shown in Figure 1, a kind of index operation method of business risk regulation engine, comprising the following steps:
S1, Spark cluster is configured on director server, the IP address of the Servers-all in addition to director server is arranged
Into the Spark cluster of director server.
Such as:
Spark1:192.168.156.101
Spark2:192.168.156.102
Spark3:192.168.156.103
Spark4:192.168.156.104
Wherein, 192.168.156.101,192.168.156.102,192.168.156.103,192.168.156.104
It is the IP address of server.
Logic control is arranged when script drive module is arranged in S2, the setting target script drive module on director server
Parameter and SparkContext, logic control parameter are managed by the Cluster that SparkContext is transferred to Spark cluster
Device.SparkContext is the api interface for connecting script drive module and Cluster manager.
Index processor active task is assigned to the Cluster manager in Spark cluster by S3, script drive module.
S4, pass through MapReduce mechanism, Cluster manager disassembles whole index processor active task.
MapReduce is a kind of distributed computing platform, is made of two stages: Map and Reduce.Map's applies
The Mapping and Converting of the one-to-one element of data is needed in us, such as is intercepted, is filtered or any conversion is grasped
Make, these one-to-one element conversions are referred to as being Map；Reduce is mainly exactly the polymerization of element, is exactly multiple elements to one
The polymerization of a element, for example seek Sum etc., here it is Reduce.
" busy extent " of S5, Cluster manager by the index processor active task after dismantling according to other servers, carry
Onto other relatively idle servers.
Specific steps are as follows:
1.Map reads global index processor active task, and global index task is parsed into < key using index as minimum unit,
Vaule>, each<key, map function of vaule>calling, such as include 10 indexs in global index task, then it is whole
Index can parse into<1, A1>,<2, A2>,<3, A1>,<4, A3>,<5, A2>,<6, A1>,<7, A2>,<8, A2>,<9, A3>,
<10,A1>；
2. cover map (), receive 1 generate<key, vaule>, be converted to new<key, vaule>output:
<A1,1>,<A1,1>,<A1,1>,<A1,1>；<A2,1>,<A2,1>,<A2,1>,<A2,1>；<A3,1>,<A3,1
>；
3. pair 2 outputs<key, vaule>be grouped.
4. being grouped according to different key values to data, the value of identical key is put into a set.After grouping
Are as follows:<A1, { 1,1,1,1 }>,<A2, { 1,1,1,1 }>,<A3, { 1,1 }>.
5.Cluster manager judge " busy extent " of other servers by multiple map tasks according to different groupings,
It is handled by network copy to other servers.
6. other server final output<A1, { 4 }>,<A2, { 4 }>,<A3, { 2 }>.
Embodiment one, Cluster manager judge the strategy of the nginx load balancing of " busy extent " of other servers
For poll distribution method, each index processor active task is assigned to other servers one by one in chronological order, if a certain server is delayed
Fall, automatic rejection, remaining continuation poll.
Embodiment two, Cluster manager judge the strategy of the nginx load balancing of " busy extent " of other servers
For Method for Weight Distribution, by monitor other servers CPU occupancy come the weight of configuration access server, specify access clothes
The probability of business device, the weight are directly proportional with access probability.
Embodiment two is upgrade method on the basis of example 1, passes through the application added in upstream parameter
Specified parameter is added after server ip can be realized, such as:
By configuring above, all index processor active tasks can all first pass through nginx Reverse Proxy, in director server
When forwarding a request to other servers, the address that upstream is tomcatsever1 is read, reads distribution policy, configuration
Tomcat1 weight is 3, so nginx will can largely request the tomcat1 being sent on 49 servers, that is, 8080 ends
Mouthful；Fewer parts realizes conditional load balancing to tomcat2.
Implementing result is transmitted to cache module and stored by S6, every server after having executed index processor active task
And it is back to director server.
Finally, it should be noted that those skilled in the art various changes and modifications can be made to the invention without departing from
The spirit and scope of the present invention.In this way, if these modifications and changes of the present invention belongs to the claims in the present invention and its waits system
Within the scope of counting, then the present invention is also intended to encompass these modification and variations.
Claims (7)
1. a kind of index operation method of business risk regulation engine, which comprises the following steps:
S1, Spark cluster is configured on director server, the IP address of the Servers-all in addition to director server is arranged to total
In the Spark cluster of server；
Logic control parameter is arranged when script drive module is arranged by S2, the setting target script drive module on director server
And SparkContext, the logic control parameter are managed by the Cluster that SparkContext is transferred to Spark cluster
Device；
Index processor active task is assigned to the Cluster manager in Spark cluster by S3, script drive module；
S4, pass through MapReduce mechanism, the Cluster manager disassembles whole index processor active task；
" busy extent " of S5, Cluster manager by the index processor active task after dismantling according to other servers, is mounted to phase
To on other idle servers；
Implementing result is transmitted to cache module and is stored and returned by S6, every server after having executed index processor active task
It is back to director server.
2. a kind of index operation method of business risk regulation engine according to claim 1, which is characterized in that step S5
In, judge that " busy extent " of other servers is judged according to Nginx load balancing.
3. a kind of index operation method of business risk regulation engine according to claim 2, which is characterized in that described
Nginx realizes that the strategy of load balancing is poll distribution method, and each index processor active task is assigned to other one by one in chronological order
Server, if a certain server is broken down, automatic rejection, remaining continuation poll.
4. a kind of index operation method of business risk regulation engine according to claim 2, which is characterized in that described
Nginx realize load balancing strategy be Method for Weight Distribution, by monitor other servers CPU occupancy come configuration access
The weight of server, specifies the probability of access server, and the weight is directly proportional with access probability.
5. a kind of index operation method of business risk regulation engine according to claim 1, which is characterized in that described slow
Storing module is cache, i.e. cache memory.
6. a kind of index operation method of business risk regulation engine according to claim 1, which is characterized in that step S5
In, the Cluster manager judges the strategy of the nginx load balancing of " busy extent " of other servers for poll distribution
Method, each index processor active task are assigned to other servers one by one in chronological order, if a certain server is broken down, pick automatically
It removes, remaining continuation poll.
7. a kind of index operation method of business risk regulation engine according to claim 1, which is characterized in that step S5
In, the Cluster manager judges the strategy of the nginx load balancing of " busy extent " of other servers for weight distribution
Method, by monitor other servers CPU occupancy come the weight of configuration access server, specify the several of access server
Rate, the weight are directly proportional with access probability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811604910.5A CN109710233A (en) | 2018-12-26 | 2018-12-26 | A kind of index operation method of business risk regulation engine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811604910.5A CN109710233A (en) | 2018-12-26 | 2018-12-26 | A kind of index operation method of business risk regulation engine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109710233A true CN109710233A (en) | 2019-05-03 |
Family
ID=66257746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811604910.5A Pending CN109710233A (en) | 2018-12-26 | 2018-12-26 | A kind of index operation method of business risk regulation engine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109710233A (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648674A (en) * | 2016-12-28 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Big data computing management method and system |
| CN107193854A (en) * | 2016-03-14 | 2017-09-22 | 商业对象软件有限公司 | Uniform client for distributed processing platform |
| US20180075107A1 (en) * | 2016-09-15 | 2018-03-15 | Oracle International Corporation | Data serialization in a distributed event processing system |
-
2018
- 2018-12-26 CN CN201811604910.5A patent/CN109710233A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107193854A (en) * | 2016-03-14 | 2017-09-22 | 商业对象软件有限公司 | Uniform client for distributed processing platform |
| US20180075107A1 (en) * | 2016-09-15 | 2018-03-15 | Oracle International Corporation | Data serialization in a distributed event processing system |
| CN106648674A (en) * | 2016-12-28 | 2017-05-10 | 北京奇艺世纪科技有限公司 | Big data computing management method and system |
Non-Patent Citations (7)
| Title |
|---|
| RICKIYANG: "hadoop学习(七)----mapReduce原理以及操作过程", 《HTTPS://WWW.CNBLOGS.COM/RICKIYANG/P/11074201.HTML》 * |
| 刘鹏: "基于Spark的数据管理平台的设计与实现", 《中国优秀博硕士学位论文全文数据库(硕士)》 * |
| 张磊等: "基于Spark的交互式数据预处理系统", 《计算机系统应用》 * |
| 林子雨: "《Spark编程基础(Scala版)》", 1 July 2018 * |
| 肖睿: "《基于Hadoop与Spark的大数据开发实战》", 1 March 2018 * |
| 赵玲玲等: "基于Spark的流程化机器学习分析方法", 《计算机系统应用》 * |
| 郑天名: "《向技术管理者转型：软件开发人员跨行业、技术、管理的转型思维与实际》", 31 October 2017 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109218355B (en) | Load balancing engine, client, distributed computing system and load balancing method | |
| US10048996B1 (en) | Predicting infrastructure failures in a data center for hosted service mitigation actions | |
| CN107832153B (en) | Hadoop cluster resource self-adaptive allocation method | |
| CA2471594C (en) | Method and apparatus for web farm traffic control | |
| US20170237647A1 (en) | Virtual network function resource allocation and management system | |
| Zhang et al. | An effective heuristic for on-line tenant placement problem in SaaS | |
| US9870269B1 (en) | Job allocation in a clustered environment | |
| WO2020036738A1 (en) | Burst performance of database queries according to query size | |
| CN107592345B (en) | Transaction current limiting device, method and transaction system | |
| US11727004B2 (en) | Context dependent execution time prediction for redirecting queries | |
| CN100440891C (en) | Method for balancing gridding load | |
| El Khoury et al. | Energy-aware placement and scheduling of network traffic flows with deadlines on virtual network functions | |
| CN107977271B (en) | Load balancing method for data center integrated management system | |
| CN112559135B (en) | Container cloud resource scheduling method based on QoS | |
| Limam et al. | Data replication strategy with satisfaction of availability, performance and tenant budget requirements | |
| CN109710413A (en) | A kind of integral Calculation Method of the rule engine system of semi-structured text data | |
| CN109032800A (en) | A kind of load equilibration scheduling method, load balancer, server and system | |
| CN113468221A (en) | System integration method based on kafka message data bus | |
| US8032636B2 (en) | Dynamically provisioning clusters of middleware appliances | |
| Yang et al. | An energy-efficient storage strategy for cloud datacenters based on variable K-coverage of a hypergraph | |
| US8812578B2 (en) | Establishing future start times for jobs to be executed in a multi-cluster environment | |
| CN114666335A (en) | DDS-based distributed system load balancing device | |
| Beigrezaei et al. | Minimizing data access latency in data grids by neighborhood‐based data replication and job scheduling | |
| CN102147887A (en) | Enterprise electronic commerce information flow management system | |
| US11888938B2 (en) | Systems and methods for optimizing distributed computing systems including server architectures and client drivers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190503 |
|
| RJ01 | Rejection of invention patent application after publication |