CN110019651A - A kind of streaming regulation engine and business data processing method - Google Patents
A kind of streaming regulation engine and business data processing method Download PDFInfo
- Publication number
- CN110019651A CN110019651A CN201910157335.7A CN201910157335A CN110019651A CN 110019651 A CN110019651 A CN 110019651A CN 201910157335 A CN201910157335 A CN 201910157335A CN 110019651 A CN110019651 A CN 110019651A
- Authority
- CN
- China
- Prior art keywords
- business rule
- data
- business
- module
- rule
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 99
- 238000000034 method Methods 0.000 claims description 15
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000006116 polymerization reaction Methods 0.000 abstract description 4
- 230000015654 memory Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 11
- 230000007246 mechanism Effects 0.000 description 10
- 238000009826 distribution Methods 0.000 description 8
- 238000003860 storage Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 6
- 230000000712 assembly Effects 0.000 description 5
- 238000000429 assembly Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000009432 framing Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000004744 fabric Substances 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000000379 polymerizing effect Effects 0.000 description 2
- 230000003936 working memory Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013497 data interchange Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
Abstract
The present invention discloses a kind of streaming regulation engine and business data processing method.The streaming regulation engine: including business rule configuration module and processing module；Wherein, the business rule configuration module is deployed in the equipment in Flink framework platform, is used for configuration service regular data；The processing module is deployed on other distributed apparatus in Flink framework platform, for obtaining the business rule data of the configuration from an equipment, the business rule data of acquisition are parsed, the business rule data parsed and business datum are subjected to rule match processing, export processing result.Technical solution provided by the invention can improve business datum polymerization speed and improve operational capability, be more suitable for user and carry out operational a large amount of converging operations, and be suitable for open source environment.
Description
Technical field
The present invention relates to computer big data technical fields, and in particular to a kind of streaming regulation engine and business data processing
Method.
Background technique
At present in the IT business system of many enterprises, a large amount of business rule configuration is often had, company manager is worked as
Decision when changing, these business rules are also required to change therewith.In order to adapt to the demand of business administration variation, enterprise
The IT business system of industry needs energy quickly and low cost is updated.In the prior art, usually by the configuration list of business rule
It solely extracts, is allowed to keep lower coupling with operation system, and realize the program of such function, be developed to as rule
Engine.
So-called regulation engine is a kind of inference engine, it is that rule are matched from rule-based knowledge base according to the existing fact
Then, it and handles in the presence of the rule to conflict, the rule passed through is finally screened in execution.Therefore, regulation engine is artificial intelligence study's neck
The a part in domain has certain selection judgement property, artificial intelligence and rich in intellectual.Meanwhile regulation engine is a kind of embedding
Enter component in the application, realize and separate operational decision making from application code, and using predefined
Semantic modules write operational decision making.Most of regulation engine all supports the order of rule and rule conflict to examine, and supports simple
The rule of scripting language is realized, the insertion of general development language is supported to develop.There are multiple regulation engines available in the industry at present,
Popular regulation engine includes business rules engines iLog and open source regulation engine Drools etc..In Drools, rule
It is stored in Production Memory (rule base), inference machine wants matched facts (fact) to be stored in Working
In Memory (working memory).After the fact is inserted into working memory, regulation engine can be the mould in true and rule base
Formula is matched, and is responsible for the rule that are excited in specific execution reasoning algorithm by Agenda (proxy server) again for the rule of successful match
Conclusion part then, at the same Agenda manage these conflict rule by conflict decision strategy execute sequence.
In the prior art, use it is most be a open source regulation engine based on Java language, its sharpest edges
It is that syntax rule is simple, similar written in Java threshold is not high, being capable of seamless process and Java is integrated and user can be to Drools
Rule carry out dynamic configuration, but the regulation engine have the shortcomings that it is obvious: due to Drools engine be mostly single machine deployment,
Operational capability is difficult to extend, therefore causes built-in polymerizable functional speed slow, is not suitable for enterprise itself or client's usage scenario
Under a large amount of converging operation tasks；In addition, its built-in sequence of events treatment mechanism is also required to consume a large amount of memory sources.Therefore,
Most of open source regulation engines in the prior art, when in face of the large-scale data that business datum amount rapidly increases, can seem nothing
It can be power.
Number of patent application be " 201310641753.6 " patent of invention in disclose a kind of distributed rule automotive engine system,
On regular distribution to multiple regulation engines, handled by multiple regulation engines, it can extended arithmetic to a certain extent
Ability, but this kind of distributed rule engine is based on Rete network, and only intra-company's research and development use, not by such as
The test of the big data quantity of the major companies such as Google, Uber, Airbnb, Amazon, Apple, Facebook and Alibaba, institute
With when big data quantity, will highlight the speed of service, scalability and in terms of performance it is poor, coordinate it is each
The carry out distributed treatment of machine is indifferent.Support of this kind of distributed rule engine due to lacking open source community, causes easily
Not strong with property, software upgrading updating speed is slow, and after going wrong, the period for repairing problem is slow, and repair ability is insufficient；And this kind point
Cloth rule engine system is to carry out collaboration processing by multiple regulation engines, since the rule of each regulation engine is different, is caused
Result data after each rules engines processes requires to carry out across a network transmission, and when data volume is big, efficiency of transmission is special
It is not low, very big pressure is caused to Rete network, and lack fault tolerant mechanism and monitoring mechanism and administrative mechanism, rule is caused to be drawn
Inaccurate coordination between holding up, it is poor to the monitoring of each regulation engine and managerial ability, each regulation engine cannot be coordinated to the money of system
The distribution in source, in fact it could happen that the resource increase that part regulation engine needs causes each regulation engine to make a mad rush for system resource, so that
The problem of entire distributed rule automotive engine system avalanche.
Therefore, be sought after it is a kind of be more suitable for open source regulation engine can solve the above problem.
Summary of the invention
In view of this, can be mentioned it is an object of the invention to propose a kind of streaming regulation engine and business data processing method
High business datum polymerization speed and raising operational capability, are more suitable for user and carry out operational a large amount of converging operations, and be suitable for
Open source environment.
According to an aspect of the present invention, a kind of streaming regulation engine is provided:
Including business rule configuration module and processing module；Wherein,
The business rule configuration module is deployed in the equipment in Flink framework platform, for configuration service rule
Data；
The processing module is deployed on other distributed apparatus in Flink framework platform, is used for from an equipment
The upper business rule data for obtaining the configuration, parse the business rule data of acquisition, the business rule that will be parsed
Data and business datum carry out rule match processing, export processing result.
Preferably, the processing module includes: that business rule obtains module, business rule parsing module, at business datum
Manage module and result output module, the business rule obtain module, business rule parsing module, business data processing module and
As a result output module is arranged on other each described distributed apparatus simultaneously, wherein other described distributed apparatus include task
Equipment where manager；
The business rule obtains module, for obtaining the business rule data of the configuration from an equipment,
Described in an equipment include equipment where work manager；
The business rule parsing module is carried out for obtaining the business rule data that module obtains to the business rule
Parsing；
The business data processing module, business rule data for parsing the business rule parsing module with
Business datum carries out rule match processing；
The result output module, for exporting the processing result of the business data processing module.
Preferably, the rule configuration page configuration service regular data that the business rule configuration module passes through setting.
Preferably, after the business rule configuration module configuration service regular data, the business rule data are used
JSON format is stored into database；
The business rule parsing module parses the business rule data of the JSON format.
Preferably, it after the business rule obtains the business rule data that module obtains the configuration, will be deemed as effectively
Business rule data change into Transmitting Data Stream to the business rule parsing module.
According to another aspect of the present invention, a kind of business data processing method is provided:
By the business rule configuration module configuration service regular data of streaming regulation engine, wherein the business rule is matched
It sets in the equipment that module is deployed in Flink framework platform；
The business rule data of the configuration are obtained from an equipment by the processing module of streaming regulation engine, it is right
The business rule data of acquisition are parsed, and the business rule data parsed and business datum are carried out rule match processing,
Processing result is exported, wherein the processing module is deployed on other distributed apparatus in Flink framework platform.
Preferably, the business rule data that the configuration is obtained from an equipment, comprising: pass through the processing
Business rule in module obtains the business rule data that module obtains the configuration from an equipment, wherein described one sets
The standby equipment including where work manager；
The business rule data of described pair of acquisition parse, comprising: pass through the business rule solution in the processing module
Analysis module obtains the business rule data that module obtains to the business rule and parses；
The business rule data that will be parsed and business datum carry out rule match processing, comprising: pass through the place
The business rule data and business number that business data processing module in reason module parses the business rule parsing module
According to progress rule match processing；
The output processing result, comprising: the business number is exported by the result output module in the processing module
According to the processing result of processing module；
Wherein the business rule obtains module, business rule parsing module, business data processing module and result output
Module is arranged on other each described distributed apparatus simultaneously, wherein other described distributed apparatus include task manager institute
Equipment.
Preferably, the rule configuration page configuration service regular data that the business rule configuration module passes through setting.
Preferably, after the business rule configuration module configuration service regular data, the business rule data are used
JSON format is stored into database；
The business rule parsing module parses the business rule data of the JSON format.
Preferably, it after the business rule obtains the business rule data that module obtains the configuration, will be deemed as effectively
Business rule data change into Transmitting Data Stream to the business rule parsing module.
Through the above it can be found that scheme provided by the embodiment of the present invention, provides a kind of new streaming rule
Engine, the streaming regulation engine include business rule configuration module and processing module；Wherein, business rule configuration module portion
It affixes one's name in the equipment in Flink framework platform, is used for configuration service regular data；The processing module is deployed in Flink frame
On other distributed apparatus in body panel, for obtaining the business rule data of the configuration from an equipment, to obtaining
The business rule data taken are parsed, and the business rule data parsed and business datum are carried out rule match processing, defeated
Processing result out.The streaming regulation engine no longer uses single machine deployment way, but the distributed deployment of Flink is utilized and divides
Business rule configuration module in streaming regulation engine is deployed in by the ability that cloth calculates by equipment group at distributed type assemblies
It is used for configuration service regular data in one equipment, processing module is deployed on other distributed apparatus, this addresses the problem biographies
Unite the CPU and the insufficient predicament of memory calculating that single machine is disposed, and is greatly strengthened by distributed computing and is gathered built in regulation engine
The capacity of conjunction ability and usable memory meets the demand of growing big data quantity and increasingly complicated business；In addition,
It also avoids being separated into multiple regulation engine collaboration processing in the prior art, and then avoids the appearance of following problems: due to each
The rule of a regulation engine is different, the result data after leading to each rules engines processes, requires to carry out across a network transmission, when
When data volume is big, efficiency of transmission is especially low, very big pressure is caused to Rete network, and lack fault tolerant mechanism and monitoring
Mechanism and administrative mechanism lead to inaccurate coordination between regulation engine, poor to the monitoring of each regulation engine and managerial ability, cannot assist
Adjust distribution of each regulation engine to the resource of system, in fact it could happen that the resource increase that part regulation engine needs leads to each rule
Then engine makes a mad rush for system resource, so that entire distributed rule automotive engine system avalanche.
Further, business rule configuration module of the invention is can to configure page configuration service by the rule of setting
Regular data, configuring the page by rule can detach business rule (service logic) from program code, so that
The configuration of business rule is more standardized and specialized.
Further, the present invention writes on business rule in Drools script not as the prior art, then converts
It at Java object and recalls and changes Java object data are operated, but business rule is changed into JSON (JavaScript
Object Notation, JS object numbered musical notation, is a kind of data interchange format of lightweight) the format storage of string is to database (example
Such as Redis) in, then data are operated again by the data in parsing JSON string, are dealt with simpler.
Detailed description of the invention
Disclosure illustrative embodiments are described in more detail in conjunction with the accompanying drawings, the disclosure above-mentioned and its
Its purpose, feature and advantage will be apparent, wherein in disclosure illustrative embodiments, identical reference label
Typically represent same parts.
Fig. 1 is an a kind of schematic diagram of the structural framing of streaming regulation engine of the embodiment of the present invention；
Fig. 2 is a kind of another schematic diagram of the structural framing of streaming regulation engine of the embodiment of the present invention；
Fig. 3 is an a kind of flow diagram of business data processing method of the embodiment of the present invention；
Fig. 4 is a kind of another flow diagram of business data processing method of the embodiment of the present invention.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with specific embodiment, and reference
Attached drawing, the present invention is described in more detail.
Although showing the preferred embodiment of the disclosure in attached drawing, however, it is to be appreciated that may be realized in various forms
The disclosure is without that should be limited by the embodiments set forth herein.On the contrary, thesing embodiments are provided so that the disclosure more
Add thorough and complete, and the scope of the present disclosure can be completely communicated to those skilled in the art.
The present invention provides a kind of streaming regulation engine, can improve business datum polymerization speed and improve operational capability, more suitable
It shares family and carries out operational a large amount of converging operations, and be suitable for open source environment.
Below in conjunction with the technical solution of attached drawing the present invention is described in detail embodiment.
Fig. 1 is an a kind of schematic diagram of the structural framing of streaming regulation engine of the embodiment of the present invention.
Shown in referring to Fig.1, a kind of streaming regulation engine of the invention, including business rule configuration module 10 and processing module
20。
Wherein, the business rule configuration module 10 is deployed in the equipment in Flink framework platform, for matching the purchase of property
Business regular data.
Wherein, the processing module 20 is deployed on other distributed apparatus in Flink framework platform, is used for from described
The business rule data that the configuration is obtained in one equipment parse the business rule data of acquisition, the industry that will be parsed
Regular data of being engaged in and business datum carry out rule match processing, export processing result.
Wherein, the business rule configuration module 10 can be the rule configuration page configuration service rule number by setting
According to.
It wherein, can also be by the business rule number after the 10 configuration service regular data of business rule configuration module
It stores according to using JSON format into database, such as storage is into Redis.
From the embodiment it can be found that the present invention provides a kind of new streaming regulation engine, the streaming regulation engine packet
Include business rule configuration module and processing module；Wherein, the business rule configuration module is deployed in Flink framework platform
In one equipment, it to be used for configuration service regular data；Other distributions that the processing module is deployed in Flink framework platform are set
It is standby upper, for obtaining the business rule data of the configuration from an equipment, the business rule data of acquisition are solved
The business rule data parsed and business datum are carried out rule match processing, export processing result by analysis.The streaming rule is drawn
It holds up no longer using single machine deployment way, but the distributed deployment of Flink and the ability of distributed computing is utilized, by equipment group
At distributed type assemblies, the business rule configuration module deployment in streaming regulation engine is used for configuration service rule on a device
Processing module is deployed on other distributed apparatus by data, and this addresses the problem the CPU of conventional individual deployment and memory to calculate
Insufficient predicament greatly strengthens the built-in polymerizing power of regulation engine and the capacity of usable memory by distributed computing,
Meet the demand of growing big data quantity and increasingly complicated business；In addition, also avoiding being separated into the prior art
Multiple regulation engine collaboration processing, and then avoid the appearance of following problems: since the rule of each regulation engine is different, cause
Result data after each rules engines processes requires to carry out across a network transmission, and when data volume is big, efficiency of transmission is special
It is not low, very big pressure is caused to Rete network, and lack fault tolerant mechanism and monitoring mechanism and administrative mechanism, rule is caused to be drawn
Inaccurate coordination between holding up, it is poor to the monitoring of each regulation engine and managerial ability, each regulation engine cannot be coordinated to the money of system
The distribution in source, in fact it could happen that the resource increase that part regulation engine needs causes each regulation engine to make a mad rush for system resource, so that
Entire distributed rule automotive engine system avalanche.Fig. 2 is a kind of the another of the structural framing of streaming regulation engine of the embodiment of the present invention
One schematic diagram.
The present invention provides one kind based on distributed real-time streaming regulation engine, is able to carry out distributed deployment, has low
Delay, high-performance, distribution, it is expansible fault-tolerant the features such as.
The present invention is based on the basis of Flink big data framework platform, the distributed deployment and distribution of Flink is utilized
The ability of calculating externally provides more equipment such as common computer composition distributed type assemblies to service, while solving biography
The CPU and the insufficient predicament of memory calculating that single machine is disposed that unite are greatly strengthened the built-in of regulation engine and are gathered by distributed computing
The capacity of conjunction ability and usable memory meets the demand of growing big data quantity and increasingly complicated business.In addition,
The present invention configures the page by rule and detaches business rule from program code, so that the configuration of business rule more mark-on
Standardization and specialization.
Referring to shown in Fig. 2, a kind of streaming regulation engine of the invention, including business rule configuration module 10 and processing module
20, wherein the processing module 20 may include: that business rule obtains module 201, business rule parsing module 202, business number
According to processing module 203 and result output module 204, wherein the business rule obtains module 201, business rule parsing module
202, other each described distributed apparatus can be arranged in business data processing module 203 and result output module 204 simultaneously
On, certainly, in other embodiments, the business rule obtains module 201, business rule parsing module 202, at business datum
Reason module 203 and result output module 204 also can be set on other distributed apparatus described in different, still, compare it
Under, at the same setting can be allowed on other each described distributed apparatus processing speed faster.
Wherein, the business rule configuration module 10 is deployed in the equipment in Flink framework platform, for matching the purchase of property
Business regular data；The processing module 20 is deployed on other distributed apparatus in Flink framework platform, is used for from described one
The business rule data that the configuration is obtained in equipment parse the business rule data of acquisition, the business that will be parsed
Regular data and business datum carry out rule match processing, export processing result.
Present invention utilizes the open source calculation blocks that Flink is an Algorithm for Distributed Data Stream Management processing and batch data processing
The characteristics of body panel.Flink is made of Jobmanager (work manager) and Taskmanager (task manager), generally
It can be deployed in respectively including a JobManager and multiple TaskManager, Jobmanager and multiple Taskmanager
On different equipment such as computer, distributed type assemblies are collectively constituted.Therefore of the invention by regulation engine and Flink framework platform
In conjunction with, the integration processing capacity of data between the modules of regulation engine is greatly strengthened, is compared, it simply will rule if being
Then each module deployment of engine on different devices, rather than in the case where being based on Flink framework platform, then needs to carry out additionally
Data Integration operation increases design and maintenance cost.
Wherein, the business rule configuration module 10 in streaming regulation engine of the invention, which can be, is deployed in an equipment
On, (business rule obtains module 201, business rule parsing module 202, business to other four modules in streaming regulation engine
Data processing module 203 and result output module 204) it can be and be deployed on other distributed apparatus, wherein the business rule
The then equipment where an equipment, that is, Jobmanager where configuration module 10, the distributed apparatus where other four modules
That is the equipment where Taskmanager, four modules all have deployment simultaneously in other each equipment.The present invention can be by
The Jobmanager of Flink is responsible for managing the deployment of four modules, four modules can also be parsed into task one by one
(task) is handled by Taskmanager, and Jobmanager and Taskmanager keep communication interaction.
Wherein, the business rule configuration module 201 can pass through the rule configuration page configuration service rule number of setting
According to.The present invention configures the page by rule and can detach business rule (service logic) from program code, so that
The configuration of business rule is more standardized and specialized.
Wherein, the business rule obtains module 202, for obtaining the business rule of the configuration from an equipment
Data.
Wherein, the business rule parsing module 203, for obtaining the business that module 202 obtains to the business rule
Regular data is parsed.After the business rule obtains the business rule data that module 202 obtains the configuration, it will be deemed as
Effective business rule data change into Transmitting Data Stream to the business rule parsing module 203.
Wherein, the business data processing module 204, the industry for parsing the business rule parsing module 203
Regular data of being engaged in and business datum carry out rule match processing.
Wherein, the result output module 205, for exporting the processing result of the business data processing module 204.
It wherein, can be by the business rule data after the 10 configuration service regular data of business rule configuration module
Using the storage of JSON format into database；The business rule parsing module 202 parses the business rule of the JSON format
Data.It is to write on business rule in Drools script in the prior art, is then converted into Java object, recalls this Java pairs
As being operated to data.The present invention does not write on business rule in Drools script, but business rule is changed into JSON
String storage, then by the data in parsing JSON string, operates data, handles simpler into Redis database.
Above-mentioned to describe a kind of streaming regulation engine of the invention in detail, corresponding introduce is drawn using the streaming rule below
The business data processing method held up.
Fig. 3 is an a kind of flow diagram of business data processing method of the embodiment of the present invention.
Referring to shown in Fig. 3, which comprises
In step 301, by the business rule configuration module configuration service regular data of streaming regulation engine, wherein institute
It states in the equipment that business rule configuration module is deployed in Flink framework platform.
Wherein, the business rule configuration module can be the rule configuration page configuration service rule number by setting
According to, after configuration service regular data, can also by the business rule data using JSON format store into database, example
As stored into Redis.
In step 302, the industry of the configuration is obtained from an equipment by the processing module of streaming regulation engine
Business regular data, parses the business rule data of acquisition, and the business rule data parsed and business datum are carried out
Rule match processing, exports processing result, wherein other distributions that the processing module is deployed in Flink framework platform are set
It is standby upper.
Wherein, an equipment includes the equipment where work manager, other described distributed apparatus include task pipe
Manage the equipment where device.
From the embodiment it can be found that streaming regulation engine applied in the method for the present invention, is no longer disposed using single machine
Mode, but the distributed deployment and the ability of distributed computing of Flink is utilized, it, will by equipment group at distributed type assemblies
Business rule configuration module deployment in streaming regulation engine is used for configuration service regular data on a device, by processing module
It is deployed on other distributed apparatus, this addresses the problem the CPU of conventional individual deployment and memory to calculate insufficient predicament, passes through
Distributed computing greatly strengthens the built-in polymerizing power of regulation engine and the capacity of usable memory, meets growing
The demand of big data quantity and increasingly complicated business, and it is suitable for open source environment.
Fig. 4 is a kind of another flow diagram of business data processing method of the embodiment of the present invention.Fig. 4 is relative to Fig. 3
Business data processing method of the invention is described in more detail.
Referring to shown in Fig. 4, which comprises
Step 401, the business rule configuration module by streaming regulation engine input each industry in the rule configuration page
The business rule data of business system, using the storage of JSON format into database.
In the step, business rule data can be stored in Redis by JSON mode.
The business rule data that input is configured in the step can be as follows but not limited to this:
{ " main business System Number ": " 123 ", " user department number ": " 122 ", " subservice system information ": [{ " sub- industry
Business System Number ": " 378 ", " business datum number ": 123, " field that adds up ": acc, " filtered fields ": " ac > 32 ", " statistics
Preceding 10 field ": " field name ", " executing sequence ": " 1 " }]
Data format:
Step 402 obtains the business rule data that module obtains the configuration by the business rule of streaming regulation engine,
It will be deemed as effective business rule data and change into Transmitting Data Stream giving business rule parsing module.
The step obtains module after Redis database acquisition business rule data by business rule, and verification judgement obtains
Whether the business rule data taken are effective, will be deemed as effective business rule data and change into stream data, and are sent to business
Rule parsing module.Wherein it is possible to which timing or not timing are sent.
Business rule, which obtains module, can extract the Customs Assigned Number in business rule data, be judged by inquiry database
Whether the Customs Assigned Number has permission processing data, if it is judged that have permission, it is determined that the business rule data are effective
Business rule data.
Wherein above-mentioned decision logic includes: to first determine whether the Customs Assigned Number has access authority to main system, next is sentenced
Disconnected whether sub-system has access authority, finally judges whether that the business datum below sub-system has access authority, works as judgement
When to there is access authority, it can determine that the business rule data are effective business rule data.It should be noted that above-mentioned
It is judged as when having access authority, can also further judges that the business datum whether there is the field that system is filled in, such as
Fruit exists, it is determined that the business rule data are effective business rule data.
Step 403, by the business rule parsing module of streaming regulation engine to the business rule of received JSON format
Data are parsed.
In the step, business rule parsing module parses business rule data according to JSON format, by business rule data
In operation system number, subservice System Number, the field of filtering, top (preceding) field, the fields parsing such as regular execution sequence
Out.Wherein, the field of filtering, top (preceding) field, regular execution sequence etc. field, is all that can be arranged inside JSON.
The resolving of the step includes: first to parse the business rule data of JSON format by format, by execution sequence
Field parses, and is ranked up by executing sequence field to all business rules, then to holding inside business rule
Row task further sorts, and is first filtered according to the field of filtering, then added up, finally seeks top (preceding) field.
Step 404 by the business data processing module of streaming regulation engine is parsed business rule parsing module
Business rule data and business datum carry out rule match processing.
In the step, business data processing module by flowing business data flow and rule configuration, (advise by the business parsed
Then data) Join (connection) operation is carried out, according to the business rule in business rule data to the business number in business data flow
According to being judged one by one, and corresponding rule match processing is carried out, ultimately produces result data.It should be noted that rule herein
Then matching treatment can use rule match processing mode in the prior art, and the present invention is simultaneously not limited.
Wherein, business data flow can be the number that consumption (acquisition) comes inside business data source Kafka (Mark reaction)
According to stream, rule configuration stream is to be initially configured from step 401 and handle the business rule data parsed by a series of.
Step 405, the processing result by the result output module outgoing traffic data processing module of streaming regulation engine.
In the step, the name that as a result output module can number the business datum in JSON data is defeated as table name
Out, it such as is output in database.
As a result the result data of the processing result of output module output, may include but be not limited to the following contents: 1) sub- industry
Business system information title；2) accumulated result；3) top result.
In conclusion the present invention constructs regulation engine using Flink framework platform, calculated using the memory of Flink
Principle successfully solves the problems, such as that built-in polymerization speed is slow, so that regulation engine provided by the invention is more suitable for user and carries out business
On a large amount of converging operations；The present invention utilizes the distributed nature of Flink framework platform, can be by thousands of computers
Memory, CPU and disk calculation resources combine, therefore successfully solve the operation band because of growing big data quantity
The problem for the calculation resources deficiency come, meets the demand of growing big data quantity and increasingly complicated business, and is applicable in
In open source environment.The present invention can also pass through regular configuration page by the rule configuration page configuration service regular data of setting
Face can detach business rule (service logic) from program code, so that the configuration of business rule is more standardized
And specialization.
Above it is described in detail according to the technique and scheme of the present invention by reference to attached drawing.
In addition, being also implemented as a kind of computer program or computer program product, the meter according to the method for the present invention
Calculation machine program or computer program product include the calculating for executing the above steps limited in the above method of the invention
Machine program code instruction.
Alternatively, the present invention can also be embodied as a kind of (or the computer-readable storage of non-transitory machinable medium
Medium or machine readable storage medium), it is stored thereon with executable code (or computer program or computer instruction code),
When the executable code (or computer program or computer instruction code) by electronic equipment (or calculate equipment, server
Deng) processor execute when, so that the processor is executed each step according to the above method of the present invention.
Those skilled in the art will also understand is that, the various example logic data in conjunction with described in disclosure herein
Block, mould data block, circuit and algorithm steps may be implemented as the combination of electronic hardware, computer software or both.
The flow chart and block diagram in the drawings show the possibility of the system and method for multiple embodiments according to the present invention realities
Existing architecture, function and operation.In this regard, each box in flowchart or block diagram can represent a modulus evidence
A part of block, program segment or code, a part of the mould data block, program segment or code include one or more for real
The executable instruction of logic function as defined in existing.It should also be noted that in some implementations as replacements, being marked in box
Function can also be occurred with being different from the sequence marked in attached drawing.For example, two continuous boxes can actually substantially simultaneously
It executes capablely, they can also be executed in the opposite order sometimes, and this depends on the function involved.It is also noted that frame
The combination of figure and/or each box in flow chart and the box in block diagram and or flow chart, can be as defined in executing
The dedicated hardware based systems of functions or operations is realized, or can be come using a combination of dedicated hardware and computer instructions
It realizes.
Various embodiments of the present invention are described above, above description is exemplary, and non-exclusive, and
It is not limited to disclosed each embodiment.Without departing from the scope and spirit of illustrated each embodiment, for this skill
Many modifications and changes are obvious for the those of ordinary skill in art field.The selection of term used herein, purport
In the principle, practical application or improvement to the technology in market for best explaining each embodiment, or make the art
Other those of ordinary skill can understand each embodiment disclosed herein.
Claims (10)
1. a kind of streaming regulation engine, it is characterised in that:
Including business rule configuration module and processing module；Wherein,
The business rule configuration module is deployed in the equipment in Flink framework platform, is used for configuration service regular data；
The processing module is deployed on other distributed apparatus in Flink framework platform, for obtaining from an equipment
The business rule data for taking the configuration parse the business rule data of acquisition, the business rule data that will be parsed
Rule match processing is carried out with business datum, exports processing result.
2. system according to claim 1, it is characterised in that:
The processing module includes: that business rule obtains module, business rule parsing module, business data processing module and result
Output module, the business rule obtain module, business rule parsing module, business data processing module and result output module
It is arranged on other each described distributed apparatus simultaneously, wherein where other described distributed apparatus include task manager
Equipment；
The business rule obtains module, for obtaining the business rule data of the configuration from an equipment, wherein institute
Stating an equipment includes the equipment where work manager；
The business rule parsing module is solved for obtaining the business rule data that module obtains to the business rule
Analysis；
The business data processing module, business rule data and business for parsing the business rule parsing module
Data carry out rule match processing；
The result output module, for exporting the processing result of the business data processing module.
3. system according to claim 1 or 2, it is characterised in that:
The rule configuration page configuration service regular data that the business rule configuration module passes through setting.
4. system according to claim 2, it is characterised in that:
After the business rule configuration module configuration service regular data, the business rule data are stored using JSON format
Into database；
The business rule parsing module parses the business rule data of the JSON format.
5. system according to claim 2, it is characterised in that:
After the business rule obtains the business rule data that module obtains the configuration, effective business rule number will be deemed as
According to changing into Transmitting Data Stream to the business rule parsing module.
6. a kind of business data processing method, it is characterised in that:
By the business rule configuration module configuration service regular data of streaming regulation engine, wherein the business rule configures mould
Block is deployed in the equipment in Flink framework platform；
The business rule data for obtaining the configuration from an equipment by the processing module of streaming regulation engine, to acquisition
Business rule data parsed, the business rule data parsed and business datum are subjected to rule match processing, output
Processing result, wherein the processing module is deployed on other distributed apparatus in Flink framework platform.
7. according to the method described in claim 6, it is characterized by:
The business rule data that the configuration is obtained from an equipment, comprising: pass through the industry in the processing module
Business rule acquisition module obtains the business rule data of the configuration from an equipment, wherein an equipment includes work
Equipment where manager；
The business rule data of described pair of acquisition parse, comprising: parse mould by the business rule in the processing module
Block obtains the business rule data that module obtains to the business rule and parses；
The business rule data that will be parsed and business datum carry out rule match processing, comprising: pass through the processing mould
The business rule data and business datum that business data processing module in block parses the business rule parsing module into
Line discipline matching treatment；
The output processing result, comprising: exported at the business datum by the result output module in the processing module
Manage the processing result of module；
Wherein the business rule obtains module, business rule parsing module, business data processing module and result output module
It is arranged on other each described distributed apparatus simultaneously, wherein where other described distributed apparatus include task manager
Equipment.
8. method according to claim 6 or 7, it is characterised in that:
The rule configuration page configuration service regular data that the business rule configuration module passes through setting.
9. according to the method described in claim 7, it is characterized by:
After the business rule configuration module configuration service regular data, the business rule data are stored using JSON format
Into database；
The business rule parsing module parses the business rule data of the JSON format.
10. according to the method described in claim 7, it is characterized by:
After the business rule obtains the business rule data that module obtains the configuration, effective business rule number will be deemed as
According to changing into Transmitting Data Stream to the business rule parsing module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910157335.7A CN110019651A (en) | 2019-03-01 | 2019-03-01 | A kind of streaming regulation engine and business data processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910157335.7A CN110019651A (en) | 2019-03-01 | 2019-03-01 | A kind of streaming regulation engine and business data processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110019651A true CN110019651A (en) | 2019-07-16 |
Family
ID=67189195
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910157335.7A Pending CN110019651A (en) | 2019-03-01 | 2019-03-01 | A kind of streaming regulation engine and business data processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110019651A (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704518A (en) * | 2019-08-26 | 2020-01-17 | 苏宁云计算有限公司 | Business data processing method and device based on Flink engine |
| CN110737631A (en) * | 2019-09-10 | 2020-01-31 | 苏宁云计算有限公司 | data analysis method and device based on Flink engine |
| CN110868324A (en) * | 2019-11-22 | 2020-03-06 | 中国建设银行股份有限公司 | Service configuration method, device, equipment and storage medium |
| CN110890983A (en) * | 2019-11-26 | 2020-03-17 | 北京杰思安全科技有限公司 | Big data-based stream processing early warning method |
| CN111240693A (en) * | 2020-01-17 | 2020-06-05 | 北京三快在线科技有限公司 | Real-time data processing method, device, equipment and storage medium |
| CN111273962A (en) * | 2020-02-14 | 2020-06-12 | 腾讯科技（深圳）有限公司 | Configuration management method, device, computer readable storage medium and computer equipment |
| CN111639101A (en) * | 2020-04-27 | 2020-09-08 | 浙江时空道宇科技有限公司 | Method, device and system for correlating rule engine system of internet of things and storage medium |
| CN111639138A (en) * | 2020-06-03 | 2020-09-08 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN112131014A (en) * | 2020-09-02 | 2020-12-25 | 广州市双照电子科技有限公司 | Decision engine system and business processing method thereof |
| CN112395338A (en) * | 2019-08-15 | 2021-02-23 | 北京国双科技有限公司 | Method and device for processing telemetering data in industrial internet platform |
| CN113269547A (en) * | 2021-05-31 | 2021-08-17 | 中国农业银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114070879A (en) * | 2021-11-26 | 2022-02-18 | 安天科技集团股份有限公司 | Data acquisition unit control method, device and related equipment |
| CN114666237A (en) * | 2022-02-25 | 2022-06-24 | 众安在线财产保险股份有限公司 | Second-level monitoring method, device and storage medium |
| CN115794445A (en) * | 2023-02-06 | 2023-03-14 | 北方健康医疗大数据科技有限公司 | Data processing method, device and equipment based on flink and regular expression |
| CN112506960B (en) * | 2020-12-17 | 2024-03-19 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103092967A (en) * | 2013-01-22 | 2013-05-08 | 交通银行股份有限公司 | Business rule decision-making method and device based on rule engine |
| CN103107895A (en) * | 2013-01-10 | 2013-05-15 | 昆山百润科技有限公司 | Billing business rule engine combined system based on configuration analysis application rules and method thereof |
| CN107220098A (en) * | 2017-06-14 | 2017-09-29 | 北京奇艺世纪科技有限公司 | The implementation method and device of regulation engine |
-
2019
- 2019-03-01 CN CN201910157335.7A patent/CN110019651A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103107895A (en) * | 2013-01-10 | 2013-05-15 | 昆山百润科技有限公司 | Billing business rule engine combined system based on configuration analysis application rules and method thereof |
| CN103092967A (en) * | 2013-01-22 | 2013-05-08 | 交通银行股份有限公司 | Business rule decision-making method and device based on rule engine |
| CN107220098A (en) * | 2017-06-14 | 2017-09-29 | 北京奇艺世纪科技有限公司 | The implementation method and device of regulation engine |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112395338A (en) * | 2019-08-15 | 2021-02-23 | 北京国双科技有限公司 | Method and device for processing telemetering data in industrial internet platform |
| CN110704518A (en) * | 2019-08-26 | 2020-01-17 | 苏宁云计算有限公司 | Business data processing method and device based on Flink engine |
| WO2021036447A1 (en) * | 2019-08-26 | 2021-03-04 | 苏宁云计算有限公司 | Flink engine-based service data processing method and apparatus |
| CN110704518B (en) * | 2019-08-26 | 2022-11-08 | 苏宁云计算有限公司 | Business data processing method and device based on Flink engine |
| CN110737631A (en) * | 2019-09-10 | 2020-01-31 | 苏宁云计算有限公司 | data analysis method and device based on Flink engine |
| CN110868324A (en) * | 2019-11-22 | 2020-03-06 | 中国建设银行股份有限公司 | Service configuration method, device, equipment and storage medium |
| CN110890983B (en) * | 2019-11-26 | 2022-04-05 | 北京杰思安全科技有限公司 | Big data-based stream processing early warning method |
| CN110890983A (en) * | 2019-11-26 | 2020-03-17 | 北京杰思安全科技有限公司 | Big data-based stream processing early warning method |
| CN111240693A (en) * | 2020-01-17 | 2020-06-05 | 北京三快在线科技有限公司 | Real-time data processing method, device, equipment and storage medium |
| CN111273962A (en) * | 2020-02-14 | 2020-06-12 | 腾讯科技（深圳）有限公司 | Configuration management method, device, computer readable storage medium and computer equipment |
| CN111639101A (en) * | 2020-04-27 | 2020-09-08 | 浙江时空道宇科技有限公司 | Method, device and system for correlating rule engine system of internet of things and storage medium |
| CN111639101B (en) * | 2020-04-27 | 2022-12-06 | 浙江时空道宇科技有限公司 | Method, device and system for correlating rule engine system of internet of things and storage medium |
| CN111639138B (en) * | 2020-06-03 | 2023-04-25 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN111639138A (en) * | 2020-06-03 | 2020-09-08 | 中国联合网络通信集团有限公司 | Data processing method, device, equipment and storage medium |
| CN112131014A (en) * | 2020-09-02 | 2020-12-25 | 广州市双照电子科技有限公司 | Decision engine system and business processing method thereof |
| CN112131014B (en) * | 2020-09-02 | 2024-01-26 | 广州市双照电子科技有限公司 | Decision engine system and business processing method thereof |
| CN112506960B (en) * | 2020-12-17 | 2024-03-19 | 青岛以萨数据技术有限公司 | Multi-model data storage method and system based on ArangoDB engine |
| CN113269547A (en) * | 2021-05-31 | 2021-08-17 | 中国农业银行股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN114070879A (en) * | 2021-11-26 | 2022-02-18 | 安天科技集团股份有限公司 | Data acquisition unit control method, device and related equipment |
| CN114070879B (en) * | 2021-11-26 | 2024-01-26 | 安天科技集团股份有限公司 | Data collector control method and device and related equipment |
| CN114666237A (en) * | 2022-02-25 | 2022-06-24 | 众安在线财产保险股份有限公司 | Second-level monitoring method, device and storage medium |
| CN114666237B (en) * | 2022-02-25 | 2023-10-31 | 众安在线财产保险股份有限公司 | Second-level monitoring method, second-level monitoring device and storage medium |
| CN115794445A (en) * | 2023-02-06 | 2023-03-14 | 北方健康医疗大数据科技有限公司 | Data processing method, device and equipment based on flink and regular expression |
| CN115794445B (en) * | 2023-02-06 | 2023-07-04 | 北方健康医疗大数据科技有限公司 | Data processing method, device and equipment based on flink and rule expression |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019651A (en) | A kind of streaming regulation engine and business data processing method | |
| US11288142B2 (en) | Recovery strategy for a stream processing system | |
| EP2831767B1 (en) | Method and system for processing data queries | |
| Schultz-Møller et al. | Distributed complex event processing with query rewriting | |
| US8141085B2 (en) | Apparatus and data structure for automatic workflow composition | |
| CN109344170B (en) | Stream data processing method, system, electronic device and readable storage medium | |
| Cerny | Aspect-oriented challenges in system integration with microservices, SOA and IoT | |
| US10581701B2 (en) | Declarative service domain federation | |
| WO2015094269A1 (en) | Hybrid flows containing a continuous flow | |
| CN104618304B (en) | Data processing method and data handling system | |
| CN108604334A (en) | Method and apparatus for autonomous services composition | |
| CN109840298A (en) | The multi information source acquisition method and system of large scale network data | |
| CN108345658A (en) | Algorithm calculates decomposing process, server and the storage medium of track | |
| CN109218060A (en) | A kind of method and device of business configuration driving flow table | |
| CN112805984A (en) | System for deploying incremental network updates | |
| Celino et al. | Semantic business process analysis. | |
| CN106406985A (en) | A distributed computing frame and a distributed computing method | |
| US11334461B2 (en) | Distributed application resource determination based on performance metrics | |
| CN106383738B (en) | Task processing method and distributed computing framework | |
| González et al. | Mmc-bpm: A domain-specific language for business processes analysis | |
| CN109597826A (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN106330556A (en) | Method and device for generating service module calling associated information | |
| Chen et al. | Resource distribution equilibrium for virtual network embedding over flexi-grid optical networks | |
| CN109669777B (en) | Industrial internet big data element demand service providing method and system | |
| Khoshnevis | An approach to variability management in service-oriented product lines |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230517 Address after: Room 101, No. 227 Gaotang Road, Tianhe District, Guangzhou City, Guangdong Province, 510000 (location: Room 601) (office only) Applicant after: Yamei Zhilian Data Technology Co.,Ltd. Address before: 510000 self compiled h, Room 201, No. 1, Hanjing Road, Tianhe District, Guangzhou, Guangdong Province Applicant before: GUANGZHOU YAME INFORMATION TECHNOLOGY Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190716 |