CN106709026A - Data processing method and data processing system - Google Patents
Data processing method and data processing system Download PDFInfo
- Publication number
- CN106709026A CN106709026A CN201611235520.6A CN201611235520A CN106709026A CN 106709026 A CN106709026 A CN 106709026A CN 201611235520 A CN201611235520 A CN 201611235520A CN 106709026 A CN106709026 A CN 106709026A
- Authority
- CN
- China
- Prior art keywords
- data
- rule
- business scenario
- engine
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/04—Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Business, Economics & Management (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- General Business, Economics & Management (AREA)
- Technology Law (AREA)
- Strategic Management (AREA)
- Marketing (AREA)
- Economics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a data processing method and a data processing system. The data processing method includes: acquiring service scenario data; according to the service scenario data, acquiring test parameters corresponding to the service scenario data; according to the test parameters, acquiring measurements corresponding to the test parameters through a flow calculation engine; sending the measurements to an alarm rule engine through a Kafka data bus, and meanwhile, calling rule statements in a database through the Kafka data bus, wherein the alarm rule engine is composed of a big-data analysis engine Flink and a rule engine CEP; abnormal alarm processing of the service scenario data is realized through the alarm rule engine and combining with the rule statements, and alarm results are acquired. With the data processing method and the data processing system, transmission of data is performed through the Kafka data bus during data transmission, the problem that data formats are inconsistent in data processing is solved, and data processing efficiency is improved.
Description
Technical field
The present invention relates to data processing field, more particularly to data processing method and system.
Background technology
Enterprise KPI Key Performance Indicator (KPI：Key Performance Indicator) it is by organization internal flow
Input, the key parameter of output end are configured, sample, calculate, analyze, and weigh a kind of target formula weight pipe of flow performance
Reason index, is the instrument for the strategic objective of enterprise being decomposed into exercisable target, is the basis of enterprise performance management.
At present, for the treatment of data, the form of data data during circulation usually disunity, because form is not united
One problem, therefore the treatment and transmission of data can be influenceed.
Therefore, defect of the prior art is, in processing procedure is carried out data transmission, because data transmission format is not united
One, the efficiency of the treatment and transmission of data, influence transmission and treatment can be influenceed.
The content of the invention
For above-mentioned technical problem, the present invention provides a kind of data processing method and system, in data transmission procedure, leads to
The total demand pairs of Mark reaction are crossed according to being transmitted, in data processing, the skimble-scamble problem of data form is solved, and improve
Data-handling efficiency.
In order to solve the above technical problems, the technical scheme that the present invention is provided is：
In a first aspect, the present invention provides a kind of data processing method, including：
Step S1, obtains business scenario data；
Step S2, according to the business scenario data, obtains the corresponding test parameter of the business scenario data；
Step S3, according to the test parameter, the corresponding measured value of the test parameter is obtained by stream calculation engine；
Step S4, the measured value is sent to alarm rule engine by Mark reaction data/address bus, while by described
Mark reaction data/address bus calls the rule statements in database, the alarm rule engine by big data analysis engine Flink and
Regulation engine CEP is constituted；
Step S5, by the alarm rule engine, with reference to the rule statements, realizes the different of the business scenario data
Normal alert process, obtains alarming result.
Data processing method of the invention, its technical scheme is：Obtain business scenario data；According to the business scenario number
According to obtaining the corresponding test parameter of the business scenario data；According to the test parameter, obtain described by stream calculation engine
The corresponding measured value of test parameter；The measured value is sent to alarm rule engine by Mark reaction data/address bus, while logical
Cross the Mark reaction data/address bus and call rule statements in database, the alarm rule engine is by big data analysis engine
Flink and regulation engine CEP is constituted；By the alarm rule engine, with reference to the rule statements, the business scenario is realized
The abnormal alarm treatment of data, obtains alarming result.
Data processing method of the invention, in data transmission procedure, is transmitted by the total demand pairs evidence of Mark reaction,
In data handling procedure, the skimble-scamble problem of data form is solved, and improve data-handling efficiency.
Further, the step S5, specially：
Alarm rule engine described in Initialize installation, and obtain the corresponding rule of the business scenario data for receiving
ID；
According to the rule ID, the corresponding rule statements of the rule ID are obtained from the database；
Abnormal judgement is carried out to the business scenario data every Preset Time, result of determination is obtained；
According to the result of determination, the abnormal alarm treatment of the business scenario data is realized, obtain alarming result.
Further, the Mark reaction data/address bus connects database by data cube computation architecture.
Further, the data cube computation architecture is java database linked system structures.
Java database linked system structures, i.e. JDBC (Java Data Base Connectivity, java databases
Connection) it is a kind of Java API for performing SQL statement, unified access can be provided for various relational databases, it is by one
Class and interface composition that group Java language is write.JDBC provides an API for standard for instrument/database development personnel,
The instrument and interface of higher level can be built accordingly, database development personnel is write database application with pure Java API
Program.There are a JDBC API, programmer need to only write that a program is just much of that with JDBC API, and it can send to associated databases
SQL is called, and enhances the efficiency and quick degree for accessing data.
Further, the database is Memsql databases.
Memsql distributed relation databases, its compatible MySQL but fast 30 times of speed, can realize 1,500,000 things per second
Business.Therefore the rule statements in the present invention are stored in this Memsql database, accelerates the processing speed of data.
Further, also include, the measured value is carried out by micro- batch loading processing by the Mark reaction data/address bus.
Micro- batch loading processing can be carried out to data, while processing more data, accelerate the processing speed of data.
Second aspect, the invention provides a kind of data handling system, including：
Data acquisition module, for obtaining business scenario data；
Test parameter module, for according to the business scenario data, obtaining the corresponding test of the business scenario data
Parameter；
Measured value module, for according to the test parameter, obtaining the test parameter by stream calculation engine corresponding
Measured value；
Data transmission blocks, for the measured value to be sent to alarm rule engine by Mark reaction data/address bus, together
When rule statements in database are called by the Mark reaction data/address bus, the alarm rule engine is drawn by big data analysis
Hold up Flink and regulation engine CEP compositions；
Data processing module, for by the alarm rule engine, with reference to the rule statements, realizes the business
The abnormal alarm treatment of scape data, obtains alarming result.
Data handling system of the invention, its technical scheme is：Data acquisition module is first passed through, business scenario number is obtained
According to；Then by test parameter module, according to the business scenario data, the corresponding test ginseng of the business scenario data is obtained
Number；Then by measured value module, according to the test parameter, the corresponding survey of the test parameter is obtained by stream calculation engine
Value；Then by data transmission blocks, the measured value is sent to alarm rule engine by Mark reaction data/address bus, together
When rule statements in database are called by the Mark reaction data/address bus, the alarm rule engine is drawn by big data analysis
Hold up Fl ink and regulation engine CEP compositions；Finally by data processing module, by the alarm rule engine, with reference to described
Rule statements, realize the abnormal alarm treatment of the business scenario data, obtain alarming result.
Data handling system of the invention, in data transmission procedure, is transmitted by the total demand pairs evidence of Mark reaction,
In data handling procedure, the skimble-scamble problem of data form is solved, and improve data-handling efficiency.
Further, the data processing module, specifically for：
Alarm rule engine described in Initialize installation, and obtain the corresponding rule of the business scenario data for receiving
ID；
According to the rule ID, the corresponding rule statements of the rule ID are obtained from the database；
Abnormal judgement is carried out to the business scenario data every Preset Time, result of determination is obtained；
According to the result of determination, the abnormal alarm treatment of the business scenario data is realized, obtain alarming result.
Further, the Mark reaction data/address bus connects database by data cube computation architecture.
Further, the data cube computation architecture is java database linked system structures.
Java database linked system structures, i.e. JDBC (Java Data Base Connectivity, java databases
Connection) it is a kind of Java API for performing SQL statement, unified access can be provided for various relational databases, it is by one
Class and interface composition that group Java language is write.JDBC provides an API for standard for instrument/database development personnel,
The instrument and interface of higher level can be built accordingly, database development personnel is write database application with pure Java API
Program.There are a JDBC API, programmer need to only write that a program is just much of that with JDBC API, and it can send to associated databases
SQL is called, and enhances the efficiency and quick degree for accessing data.
Brief description of the drawings
In order to illustrate more clearly of the specific embodiment of the invention or technical scheme of the prior art, below will be to specific
The accompanying drawing to be used needed for implementation method or description of the prior art is briefly described.
Fig. 1 shows a kind of flow chart of data processing method that the embodiment of the present invention is provided；
Fig. 2 shows a kind of schematic diagram of data handling system that the embodiment of the present invention is provided.
Specific embodiment
The embodiment of technical solution of the present invention is described in detail below in conjunction with accompanying drawing.Following examples are only used for
Technical scheme is clearly illustrated, therefore is intended only as example, and protection of the invention can not be limited with this
Scope.
Embodiment one
Fig. 1 shows a kind of flow chart of data processing method that the embodiment of the present invention is provided；As shown in figure 1, this hair
Bright embodiment one provides a kind of data processing method, including：
Step S1, obtains business scenario data；
Step S2, according to business scenario data, obtains the corresponding test parameter of business scenario data；
Step S3, according to test parameter, the corresponding measured value of test parameter is obtained by stream calculation engine；
Step S4, measured value is sent to alarm rule engine by Mark reaction data/address bus, while passing through Mark reaction number
The rule statements in database are called according to bus, alarm rule engine is by big data analysis engine Flink and regulation engine CEP groups
Into；
Step S5, by alarm rule engine, binding rule sentence realizes the abnormal alarm treatment of business scenario data,
Obtain alarming result：
Specially：
Initialize installation alarm rule engine, and obtain the corresponding rule ID of business scenario data for receiving；
According to rule ID, the corresponding rule statements of rule ID are obtained from database；
Abnormal judgement is carried out to business scenario data every Preset Time, result of determination is obtained；
According to result of determination, the abnormal alarm treatment of business scenario data is realized, obtain alarming result.
Data processing method of the invention, its technical scheme is：Obtain business scenario data；According to business scenario data,
Obtain the corresponding test parameter of business scenario data；The corresponding test data of different business scenarios is different, is surveyed obtained from entering
Value is also different, therefore, according to test parameter, the corresponding measured value of test parameter is obtained by stream calculation engine；
With reference to according to measured value, data are further processed, by measured value by Mark reaction data/address bus send to
Alarm rule engine, then calls the rule statements in database, detailed process to be by Mark reaction data/address bus：First initialize
Alarm rule engine is set, and obtains the corresponding rule ID of business scenario data for receiving；Rule ID is obtained from database
Corresponding rule statements；Business scenario data are carried out with abnormal judgement every Preset Time, specific basis for estimation is：Work as rule
The corresponding rule statements of ID have altered, and judge that result of determination is target data exception；When the corresponding rule statements of rule ID do not have
Change, judges that result of determination is normal as target data.
And then result of determination is obtained, and the abnormal alarm treatment of business scenario data is realized, obtain alarming result.
Wherein, alarm rule engine is made up of big data analysis engine Flink and regulation engine CEP.
Data processing method of the invention, in data transmission procedure, is transmitted by the total demand pairs evidence of Mark reaction,
In data handling procedure, the skimble-scamble problem of data form is solved, and improve data-handling efficiency.
Preferably, Mark reaction data/address bus connects database by data cube computation architecture.
Preferably, data cube computation architecture is java database linked system structures.
Java database linked system structures, i.e. JDBC (Java Data Base Connectivity, java databases
Connection) it is a kind of Java API for performing SQL statement, unified access can be provided for various relational databases, it is by one
Class and interface composition that group Java language is write.JDBC provides an API for standard for instrument/database development personnel,
The instrument and interface of higher level can be built accordingly, database development personnel is write database application with pure Java API
Program.
Due to Java have it is firm, safe, easy to use, should be readily appreciated that and the characteristic such as can automatically download from network, be
Write the outstanding language of database application.Required is simply carried out between java application and various disparate databases
The method of dialogue.And JDBC is exactly as the mechanism of this kind of purposes.
There are JDBC API, it is not necessary to specially write a program to access sybase database, to access Oracle data
A program is specially write in storehouse again, or writes another program etc. again to access informix database, and programmer only needs to use
JDBC API write a program just it is much of that, it can to associated databases send SQL call.Meanwhile, Java language and JDBC are tied
Programmer is not necessarily different platforms altogether and write different application programs, need only write a program can just allow it in office
Run on what platform, enhance the efficiency and quick degree for accessing data.
Preferably, database is Memsql databases.
Memsql distributed relation databases, its compatible MySQL but fast 30 times of speed, can realize 1,500,000 things per second
Business.Therefore the rule statements in the present invention are stored in this Memsql database, accelerates the processing speed of data.Memsql points
The principle of cloth relevant database is only to be pre-compiled as C++ with internal memory and by SQL.
Preferably, also include, measured value is carried out by micro- batch loading processing by Mark reaction data/address bus.
Micro- batch loading processing can be carried out to data, while processing more data, accelerate the processing speed of data.Its
In, the micro- batch loading processing for carrying out data can be realized by ARM microprocessor.
Fig. 2 shows a kind of schematic diagram of data handling system that the embodiment of the present invention is provided, as shown in Fig. 2 this hair
Bright embodiment provides a kind of data handling system 10, including：
Data acquisition module 101, for obtaining business scenario data；
Test parameter module 102, for according to business scenario data, obtaining the corresponding test parameter of business scenario data；
Measured value module 103, for according to test parameter, the corresponding measurement of test parameter being obtained by stream calculation engine
Value；
Data transmission blocks 104, for measured value to be sent to alarm rule engine by Mark reaction data/address bus, while
Rule statements in database are called by Mark reaction data/address bus, alarm rule engine by big data analysis engine Flink and
Regulation engine CEP is constituted；
Data processing module 105, for by alarm rule engine, binding rule sentence to realize business scenario data
Abnormal alarm treatment, obtains alarming result；
Specially：
Initialize installation alarm rule engine, and obtain the corresponding rule ID of business scenario data for receiving；
According to rule ID, the corresponding rule statements of rule ID are obtained from database；
Abnormal judgement is carried out to business scenario data every Preset Time, result of determination is obtained；
According to result of determination, the abnormal alarm treatment of business scenario data is realized, obtain alarming result.
Data handling system of the invention 10, its technical scheme is：Data acquisition module 101 is first passed through, business is obtained
Scape data；Then by test parameter module 102, according to business scenario data, the corresponding test ginseng of business scenario data is obtained
Number；The corresponding test data of different business scenarios is different, and measured value obtained from entering is also different, therefore, by measured value mould
Block 103, according to test parameter, the corresponding measured value of test parameter is obtained by stream calculation engine；
With reference to according to measured value, data are further processed, by data transmission blocks 104, measured value is passed through
Mark reaction data/address bus is sent to alarm rule engine, and the regular language in database is then called by Mark reaction data/address bus
Sentence, then by data processing module 105, the abnormal alarm treatment of data is carried out, detailed process is：First Initialize installation alarm rule
Then engine, and the corresponding rule ID of business scenario data that acquisition is received；The corresponding rule of rule ID is obtained from database
Sentence；Business scenario data are carried out with abnormal judgement every Preset Time, specific basis for estimation is：When the corresponding rule of rule ID
Then sentence has altered, and judges that result of determination is target data exception；When the corresponding rule statements of rule ID are not changed, judge
Result is determined for target data is normal.
And then result of determination is obtained, and the abnormal alarm treatment of business scenario data is realized, obtain alarming result.
Wherein, alarm rule engine is made up of big data analysis engine Flink and regulation engine CEP.
Data handling system of the invention 10, in data transmission procedure, is transmitted by the total demand pairs evidence of Mark reaction,
In data processing, the skimble-scamble problem of data form is solved, and improves data-handling efficiency.
Preferably, Mark reaction data/address bus connects database by data cube computation architecture.
Preferably, data cube computation architecture is java database linked system structures.
Java database linked system structures, i.e. JDBC (Java Data Base Connectivity, java databases
Connection) it is a kind of Java API for performing SQL statement, unified access can be provided for various relational databases, it is by one
Class and interface composition that group Java language is write.JDBC provides an API for standard for instrument/database development personnel,
The instrument and interface of higher level can be built accordingly, database development personnel is write database application with pure Java API
Program.
Due to Java have it is firm, safe, easy to use, should be readily appreciated that and the characteristic such as can automatically download from network, be
Write the outstanding language of database application.Required is simply carried out between java application and various disparate databases
The method of dialogue.And JDBC is exactly as the mechanism of this kind of purposes.
There are JDBC API, it is not necessary to specially write a program to access sybase database, to access Oracle data
A program is specially write in storehouse again, or writes another program etc. again to access informix database, and programmer only needs to use
JDBC API write a program just it is much of that, it can to associated databases send SQL call.Meanwhile, Java language and JDBC are tied
Programmer is not necessarily different platforms altogether and write different application programs, need only write a program can just allow it in office
Run on what platform, enhance the efficiency and quick degree for accessing data.
Preferably, database is Memsql databases.
Memsql distributed relation databases, its compatible MySQL but fast 30 times of speed, can realize 1,500,000 things per second
Business.Therefore the rule statements in the present invention are stored in this Memsql database, accelerates the processing speed of data.Memsql points
The principle of cloth relevant database is only to be pre-compiled as C++ with internal memory and by SQL.
Preferably, also include, measured value is carried out by micro- batch loading processing by Mark reaction data/address bus.
Micro- batch loading processing can be carried out to data, while processing more data, accelerate the processing speed of data.Its
In, the micro- batch loading processing for carrying out data can be realized by ARM microprocessor.
Embodiment two
Based on the data processing method in embodiment one, and data handling system 10, with reference to specific development language environment
And application scenarios, carry out specific data handling procedure explanation.
Data processing method and device that the present invention is provided, development language environment are Java.
1st, application scenarios are suppressed to draw and lift, and correlation investigates dimension combination (test parameter) and is：Key Distribution values, status information,
Window operation, business hours, complex logic；Then test value is obtained according to these test parameters, corresponding test value is passed through
Mark reaction bus transfer carries out the alert process of data exception, output alarm to flow data engine big data analysis engine Flink
As a result.
Wherein, in data processing, each program called is：
Specify and be using the business hours：
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
Specify the method for extracting the business hours (preset interval time)：
assignTimestampsAndWatermarks(new UserDefineTimeAssigner())
Specify the key for being grouped：
keyBy(new UserDefineKeySelector())
Specify time window：
timeWindow(windowTimeSize,slideTimeSize)
Realize that drawing and lifting on every security suppresses logic：
apply(new LTDYWindowFunction())
Drawn and lifted in prototype system and suppress logic in two sub-sections：
1), basic alarm portion, this part passes through repeatedly to travel through the conclusion of the business sequence of same branch security, it is possible to gradually filter
A. in window, in the exceeded b. windows of price range, in investor's dealing exceeded c. windows of total value, the exceeded d. of investor's dealing ratio
In window, points are drawn and lifted exceeded.The content of basis alarm is just obtained by above-mentioned 4 step, can be completed on this basis
2), reversely conclude the business alarm portion, a. using basis alarm as state storage in udf, b. subsequent flowings come
Data update the state in udf, and count anti-phase transaction aggregate-value, and c. is calculated when each window is calculated and is triggered and reversely handed over
Easily alarm.
2nd, application scenarios are that wholesale envelope rises, and correlation investigates dimension combination (test parameter) and is：Key Distribution values, status information,
Complex logic；Then test value is obtained according to these test parameters, by corresponding test value by Mark reaction bus transfer to stream
Data engine big data analysis engine Flink, carries out the alert process of data exception, exports alarming result.
Wherein, in data processing, each program called is：
Specify the key for being grouped：
keyBy(new UserDefineKeySelector())
Realize the wholesale envelope limit-up logic on every security：
flatMap(new DEFZTFlatMapFunction())
Can record the relevant information of all limit-ups (limit down) security in DEFZTFlatMapFunction, including each supervision is right
The commission amount of elephant；And supervised entities' commission amount and accounting meet regular trigger condition, record the time started.
For the security (there is the securities record in DEFZTFlatMapFunction) of limit-up, only two kinds situations
The lower commission amount that can update supervised entities：
1), supervised entities are bought with the commission of limit-up valency；Now increase the commission amount of correspondence supervised entities, if the supervised entities
Regular trigger condition (the commission amount of money is unsatisfactory for before>Ten thousand yuan of a1, effectively entrust accounting>=b1%), then need to check that the supervision is right
As now whether meeting regular trigger condition, if meeting the record time started；
Because the commission of securities total amount increased, the accounting of other supervised entities can be reduced, therefore which need to check
Whether the accounting for meeting the supervised entities of rule remains unchanged>=b1%, if<B1%, then remove its record time.
2), conclusion of the business during limit-up；The commission amount of passive side is now reduced, if having met rule before the supervised entities
Trigger condition, then, if be unsatisfactory for, need to remove its record time from new inspection.
Because the commission of securities total amount is reduced, the accounting of other supervised entities can increase, therefore which need to check
Meet the commission amount of money>Whether the accounting of the supervised entities that ten thousand yuan of a1>=b1%, if>=b1%, then record the time started.
When commission of securities amount has variation, (current time-supervised entities' rule triggering is opened for the inspection of triggering Rule Duration
Time beginning), if the threshold value given more than rule, exports alarm content.
For the security (not existing the securities record in DEFZTFlatMapFunction) also without limit-up, then monitoring is entrusted
Order is bought in support order, the commission if there is limit-up valency, then the security are possible to that limit-up can be arrived, and records its state, and tracking should
The instant conclusion of the business order that pen commission triggers, and the commission amount of supervised entities is reduced according to conclusion of the business order；If the pen entrusts order
There is surplus, then illustrate that the security have been pulled to limit-up, limit-up failure is drawn in otherwise explanation, removes the security state, and treatment is patrolled during limit down
Collect similar.
3rd, application scenarios be that x minute index amplitude be exceeded, timesharing ups and downs exception in disk, correlation investigates dimension combination and (tests
Parameter) be：Window operation, business hours, summary responses；Then test value is obtained according to these test parameters, by corresponding survey
Examination value, to flow data engine big data analysis engine Flink, carries out the alert process of data exception by Mark reaction bus transfer,
Output alarming result.
4th, application scenarios be that x minute member's amount of money is exceeded, timesharing ups and downs exception in disk, correlation investigates dimension combination and (tests
Parameter) be：Key Distribution values, window operation, business hours；Then test value is obtained according to these test parameters, by corresponding survey
Examination value, to flow data engine big data analysis engine Flink, carries out the alert process of data exception by Mark reaction bus transfer,
Output alarming result.
Finally it should be noted that：Various embodiments above is merely illustrative of the technical solution of the present invention, rather than its limitations；To the greatest extent
Pipe has been described in detail with reference to foregoing embodiments to the present invention, it will be understood by those within the art that：Its according to
The technical scheme described in foregoing embodiments can so be modified, or which part or all technical characteristic are entered
Row equivalent；And these modifications or replacement, the essence of appropriate technical solution is departed from various embodiments of the present invention technology
The scope of scheme, it all should cover in the middle of the scope of claim of the invention and specification.
Claims (10)
1. data processing method, it is characterised in that including：
Step S1, obtains business scenario data；
Step S2, according to the business scenario data, obtains the corresponding test parameter of the business scenario data；
Step S3, according to the test parameter, the corresponding measured value of the test parameter is obtained by stream calculation engine；
Step S4, the measured value is sent to alarm rule engine by Mark reaction data/address bus, while passing through the Kraft
Card data/address bus calls the rule statements in database, and the alarm rule engine is by big data analysis engine Flink and rule
Engine CEP is constituted；
Step S5, by the alarm rule engine, with reference to the rule statements, realizes the abnormal report of the business scenario data
Alert treatment, obtains alarming result.
2. data processing method according to claim 1, it is characterised in that
The step S5, specially：
Alarm rule engine described in Initialize installation, and obtain the corresponding rule ID of the business scenario data for receiving；
According to the rule ID, the corresponding rule statements of the rule ID are obtained from the database；
Abnormal judgement is carried out to the business scenario data every Preset Time, result of determination is obtained；
According to the result of determination, the abnormal alarm treatment of the business scenario data is realized, obtain alarming result.
3. data processing method according to claim 1, it is characterised in that
The Mark reaction data/address bus connects database by data cube computation architecture.
4. data processing method according to claim 3, it is characterised in that
The data cube computation architecture is java database linked system structures.
5. data processing method according to claim 1, it is characterised in that
The database is Memsql databases.
6. data processing method according to claim 1, it is characterised in that
Also include, the measured value is carried out by micro- batch loading processing by the Mark reaction data/address bus.
7. data handling system, it is characterised in that including：
Data acquisition module, for obtaining business scenario data；
Test parameter module, for according to the business scenario data, obtaining the corresponding test parameter of the business scenario data；
Measured value module, for according to the test parameter, the corresponding measurement of the test parameter being obtained by stream calculation engine
Value；
Data transmission blocks, for the measured value to be sent to alarm rule engine by Mark reaction data/address bus, while logical
Cross the Mark reaction data/address bus and call rule statements in database, the alarm rule engine is by big data analysis engine
Flink and regulation engine CEP is constituted；
Data processing module, for by the alarm rule engine, with reference to the rule statements, realizes the business scenario number
According to abnormal alarm process, obtain alarming result.
8. data handling system according to claim 7, it is characterised in that
The data processing module, specifically for：
Alarm rule engine described in Initialize installation, and obtain the corresponding rule ID of the business scenario data for receiving；
According to the rule ID, the corresponding rule statements of the rule ID are obtained from the database；
Abnormal judgement is carried out to the business scenario data every Preset Time, result of determination is obtained；
According to the result of determination, the abnormal alarm treatment of the business scenario data is realized, obtain alarming result.
9. data handling system according to claim 7, it is characterised in that
The Mark reaction data/address bus connects database by data cube computation architecture.
10. data handling system according to claim 9, it is characterised in that
The data cube computation architecture is java database linked system structures.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611235520.6A CN106709026A (en) | 2016-12-28 | 2016-12-28 | Data processing method and data processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611235520.6A CN106709026A (en) | 2016-12-28 | 2016-12-28 | Data processing method and data processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106709026A true CN106709026A (en) | 2017-05-24 |
Family
ID=58903599
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611235520.6A Pending CN106709026A (en) | 2016-12-28 | 2016-12-28 | Data processing method and data processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106709026A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710231A (en) * | 2018-12-21 | 2019-05-03 | 成都四方伟业软件股份有限公司 | A kind of visualization script edit methods and device |
| CN110414242A (en) * | 2019-08-02 | 2019-11-05 | 中国工商银行股份有限公司 | For detecting the method, apparatus, equipment and medium of service logic loophole |
| CN111143334A (en) * | 2019-11-13 | 2020-05-12 | 深圳市华傲数据技术有限公司 | Data quality closed-loop control method |
| CN111444291A (en) * | 2020-03-27 | 2020-07-24 | 上海爱数信息技术股份有限公司 | Real-time data alarm method based on stream processing engine and rule engine |
| CN118631687A (en) * | 2024-08-14 | 2024-09-10 | 浙江万里扬股份有限公司杭州分公司 | CAN communication message level testing method and system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544074A (en) * | 2012-07-09 | 2014-01-29 | 阿里巴巴集团控股有限公司 | Method and device for verifying service |
| CN105653425A (en) * | 2015-12-28 | 2016-06-08 | 中国民航信息网络股份有限公司 | Complicated event processing engine based monitoring system |
| CN106101090A (en) * | 2016-06-07 | 2016-11-09 | 中国建设银行股份有限公司 | Operational approach and rule engine system for regulation engine |
-
2016
- 2016-12-28 CN CN201611235520.6A patent/CN106709026A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103544074A (en) * | 2012-07-09 | 2014-01-29 | 阿里巴巴集团控股有限公司 | Method and device for verifying service |
| CN105653425A (en) * | 2015-12-28 | 2016-06-08 | 中国民航信息网络股份有限公司 | Complicated event processing engine based monitoring system |
| CN106101090A (en) * | 2016-06-07 | 2016-11-09 | 中国建设银行股份有限公司 | Operational approach and rule engine system for regulation engine |
Non-Patent Citations (1)
| Title |
|---|
| TANMAY DESHPANDE: ""Complex Event Processing (CEP) on disparate, high frequency data streams using Apache Flink and Kafka"", 《RESEARCHGATE》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710231A (en) * | 2018-12-21 | 2019-05-03 | 成都四方伟业软件股份有限公司 | A kind of visualization script edit methods and device |
| CN110414242A (en) * | 2019-08-02 | 2019-11-05 | 中国工商银行股份有限公司 | For detecting the method, apparatus, equipment and medium of service logic loophole |
| CN111143334A (en) * | 2019-11-13 | 2020-05-12 | 深圳市华傲数据技术有限公司 | Data quality closed-loop control method |
| CN111444291A (en) * | 2020-03-27 | 2020-07-24 | 上海爱数信息技术股份有限公司 | Real-time data alarm method based on stream processing engine and rule engine |
| CN118631687A (en) * | 2024-08-14 | 2024-09-10 | 浙江万里扬股份有限公司杭州分公司 | CAN communication message level testing method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106709026A (en) | Data processing method and data processing system | |
| CN107958049A (en) | A kind of quality of data checking and administration system | |
| CN107810500A (en) | Data quality analysis | |
| CN109542789B (en) | Code coverage rate statistical method and device | |
| CN107992401A (en) | Performance test evaluation method, device, terminal device and storage medium | |
| WO2017167022A1 (en) | Big data-based method and device for calculating relationship between development objects | |
| Bischofberger et al. | Sotograph–a pragmatic approach to source code architecture conformance checking | |
| CN112181955B (en) | Data standard management method for information sharing of heavy haul railway comprehensive big data platform | |
| CN106502880A (en) | A kind of memory overflow adjustment method and device | |
| CN106776102A (en) | A kind of application system health examination method and system | |
| CN105183542B (en) | A kind of EMS memory management process and system | |
| CN108897686A (en) | It is complete to record separately automated testing method and device | |
| CN114511235A (en) | Process evaluation method and system | |
| CN110471912B (en) | Employee attribute information verification method and device and terminal equipment | |
| CN108845941A (en) | A kind of SQL injection test coverage statistical method and system | |
| CN112825165A (en) | Project quality management method and device | |
| CN112446601B (en) | Method and system for diagnosing data of uncomputable area | |
| CN117668733A (en) | Knowledge graph-based abnormal data diagnosis method, device, equipment and medium | |
| CN107644041A (en) | Declaration form settlement processing method and device | |
| CN113032254B (en) | Test coverage condition evaluation method and device | |
| CN113641628B (en) | Data quality detection method, device, equipment and storage medium | |
| CN111078783A (en) | Data management visualization method based on supervision and protection | |
| CN107273293B (en) | Big data system performance test method and device and electronic equipment | |
| CN109947828A (en) | A kind of method and apparatus handling report data | |
| CN108519949A (en) | A kind of test method, system and the device of ORACLE databases pressure |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170524 |
|
| RJ01 | Rejection of invention patent application after publication |