CN105005525A - Middleware based service data monitoring method and system - Google Patents
Middleware based service data monitoring method and system Download PDFInfo
- Publication number
- CN105005525A CN105005525A CN201510505642.1A CN201510505642A CN105005525A CN 105005525 A CN105005525 A CN 105005525A CN 201510505642 A CN201510505642 A CN 201510505642A CN 105005525 A CN105005525 A CN 105005525A
- Authority
- CN
- China
- Prior art keywords
- data
- business datum
- business
- middleware
- supervisory system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a middleware based service data monitoring method and system. The method comprises steps as follows: service data from different application systems are sent to a monitoring system through a middleware to be monitored; the service data are stored and subjected to modeling according to preset storage rules; the service data subjected to storage and modeling processing are classified; a service correlation model is established according to data classification, and each service datum is subjected to trajectory tracking; a pre-warning threshold value is set according to the service correlation model, and accordingly, the service data are monitored. By means of the technical scheme, the different service data can be intelligently monitored, the universality as well as the adaptability to change of the service data and the service rules is effectively improved, and the monitoring demand of most industries can be rapidly met.
Description
Technical field
The present invention relates to data stream monitoring technical field, in particular to a kind of business datum method for supervising based on middleware and a kind of business datum supervisory system based on middleware.
Background technology
Current computer IT field, the significant components of software based on middleware, advances side by side with operating system, database, worldwide presents the impetus of fast development, has formed a huge industry.Middleware should be one of market with the fastest developing speed at home in whole Software Industry.And the monitoring technique of current traditional middleware, comprise monitoring cable number of passes, processing time, number of request, byte number, Cluster (cluster), storehouse, thread pool, connection pool, Web (World Wide Web, WWW) index parameter such as application grade, these monitoring techniques are more common, applicable industry is also more extensive, but the monitoring mostly just to performance, without the monitoring capacity of business data flow, and often real valuable monitoring is all the monitoring to business datum.Also some monitoring techniques based on middleware are had to be for business datum in current industry, such as, BAM (BusinessActivity Monitoring: BAM) technology in Microsoft BizTalk (Business Talk: commercial session), but versatility is lacked mostly to the monitoring of business data flow and needs too much manual intervention, these method for supervising need the data model knowing monitoring business, and monitor according to customed rule, thus these monitoring techniques can not widely use, and changeableness is lacked to the change of data and the change of business rule, the monitoring requirement of large conglomerate can not be met fast.
Therefore, need a kind of new technical scheme, the intelligent monitoring to different business data stream can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast, become problem demanding prompt solution.
Summary of the invention
The present invention is just based on the problems referred to above, propose a kind of new technical scheme, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast.
In view of this, the present invention proposes a kind of business datum method for supervising based on middleware, comprising: the business datum from different application systems is sent to supervisory system by middleware, to monitor described business datum; According to default storage rule, data storage and data modeling are carried out to described business datum; Data classification is carried out to the described business datum through described data storage and described data modeling process; Service correlation model is set up, to carry out track following to business datum described in every bar according to described Data classification; According to described service correlation model, threshold value of warning is set, to monitor described business datum.
In this technical scheme, various different application system is connected by middleware, the business datum of different application systems is sent to supervisory system, and stored by data, data modeling, and can the ability of not designated data structure in advance in conjunction with NoSQL database, the structure of data model is designed to dynamic change, oneself can increase in actual O&M process to make model, self-teaching, then Data classification is passed through, set up service correlation model, monitoring and early warning threshold value is set, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast, learn by using large data teaching display stand, and pass through intelligent modeling, intelligent classification, intelligent predicting early warning reaches the target of monitoring business data.
In technique scheme, preferably, before described business datum is sent to described supervisory system by described middleware, described business datum is judged whether to be sent to described supervisory system, and when judged result is for being, after described business datum being converted to XML data by adapter, be sent to described supervisory system; And according to described default storage rule, based on NoSQL database, the storage of described data and described data modeling are carried out to described business datum.
In this technical scheme, before business datum is sent to supervisory system by middleware, that is, by business datum access or can select when being sent to corresponding application system to monitor the need of to this business datum, and when needs are monitored, by adapter, business datum is converted to XML (Extensible Markup Language, markup language) data, to realize the mark to needing the business datum monitored, and based on NoSQL (Not Only StructuredQuery Language, being not only Structured Query Language (SQL)) database Cassandra (distributed NoSQL Database Systems of increasing income) carries out data storage and modeling, so that realize the storage to business datum, modeling, classification, trajectory track etc., and then the monitoring realized business datum.
In technique scheme, preferably, according to described default storage rule, described XML data is stored in described NoSQL database Cassandra, and carries out described data modeling; Calculate the weighted value of the field of XML data described in every bar, and according to described weighted value, described Data classification is carried out to described XML data.
In this technical scheme, by default storage rule, XML data is stored in NoSQL database, to realize the process to super amount data, and carries out data modeling, the data of storage can be made to have standardization, be convenient to follow-uply search and/or monitor; By calculating the weighted value of data field, intelligent classification is carried out to data, for realizing providing favourable prerequisite guarantee to the monitoring of business datum.
In technique scheme, preferably, according to described service correlation model, the probability matrix that the business that is associated occurs, to determine the inevitable business in described associated services, the threshold value of warning of other business in associated services according to described inevitable business setting, to monitor described business datum.
In this technical scheme, according to service correlation model, list the probability matrix that all related services occur, thus find the inevitable business in related service, and monitoring and early warning is carried out to the data violating inevitable business, namely realize the monitoring and early warning to different business; Also set up matrix model to the different field of same business, find out the certain event that analog value appears in different field, then coordinate alteration ruler, the data reached violating certain event carry out monitoring and early warning; To single business carry out with second, point, time, day, week, month, year for periods dimension, set up discrete fourier model, judge service period, look-ahead and monitoring and early warning are carried out to the data of single business; Namely threshold value of warning being set, when a threshold is exceeded, then carrying out early warning, realize the monitoring to business datum, so, by arranging early warning threshold values accurately, can monitor more accurately business datum.
In technique scheme, preferably, by described middleware, the interface mode that described business datum is sent to described supervisory system is comprised: File mode and/or Http mode.
In this technical scheme, supervisory system includes but not limited to File mode and/or Http mode for the interface mode that middleware system provides, namely File mode can be passed through: specified file path, filename expression formula (comprising Business Name, generation time, GUID unique identifying number), and Http mode: transmission XML data, business datum is entered in supervisory system by named variable title (comprising Business Name, generation time, GUID unique identifying number).
According to a further aspect in the invention, also proposed a kind of business datum supervisory system based on middleware, comprise: data conversion module, for the business datum from different application systems is sent to supervisory system by middleware, to monitor described business datum; Data memory module, for carrying out data storage and data modeling according to default storage rule to described business datum; Data categorization module, for carrying out Data classification to the described business datum through described data storage and described data modeling process; Business model module, for setting up service correlation model according to described Data classification, to carry out track following to business datum described in every bar; Data monitoring module, for according to described service correlation model, arranges threshold value of warning, to monitor described business datum.
In this technical scheme, various different application system is connected by middleware, the business datum of different application systems is sent to supervisory system, and stored by data, data modeling, and can the ability of not designated data structure in advance in conjunction with NoSQL database, the structure of data model is designed to dynamic change, oneself can increase in actual O&M process to make model, self-teaching, then Data classification is passed through, set up service correlation model, monitoring and early warning threshold value is set, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast, learn by using large data teaching display stand, and pass through intelligent modeling, intelligent classification, intelligent predicting early warning reaches the target of monitoring business data.
In technique scheme, preferably, also comprise: judge module, for before described business datum is sent to described supervisory system by described middleware, judge whether described business datum to be sent to described supervisory system; Described data conversion module specifically for: when judged result is for being, after described business datum being converted to XML data by adapter, be sent to described supervisory system; And described data memory module specifically for: according to described default storage rule to described business datum based on NoSQL database carry out described data store and described data modeling.
In this technical scheme, before business datum is sent to supervisory system by middleware, that is, by business datum access or can select when being sent to corresponding application system to monitor the need of to this business datum, and when needs are monitored, by adapter, business datum is converted to XML (Extensible Markup Language, markup language) data, to realize the mark to needing the business datum monitored, and based on NoSQL (Not Only StructuredQuery Language, being not only Structured Query Language (SQL)) database Cassandra (distributed NoSQL Database Systems of increasing income) carries out data storage and modeling, so that realize the storage to business datum, modeling, classification, trajectory track etc., and then the monitoring realized business datum.
In technique scheme, preferably, described data memory module specifically for: according to described default storage rule, described XML data is stored in described NoSQL database Cassandra, and carries out described data modeling; And described data categorization module is specifically for the weighted value calculating the field of XML data described in every bar, and according to described weighted value, described Data classification is carried out to described XML data.
In this technical scheme, by default storage rule, XML data is stored in NoSQL database, to realize the process to super amount data, and carries out data modeling, the data of storage can be made to have standardization, be convenient to follow-uply search and/or monitor; By calculating the weighted value of data field, intelligent classification is carried out to data, for realizing providing favourable prerequisite guarantee to the monitoring of business datum.
In technique scheme, preferably, described data monitoring module specifically for: according to described service correlation model, the probability matrix that the business that is associated occurs, to determine the inevitable business in described associated services, the threshold value of warning of other business in associated services according to described inevitable business setting, to monitor described business datum.
In this technical scheme, according to service correlation model, list the probability matrix that all related services occur, thus find the inevitable business in related service, and monitoring and early warning is carried out to the data violating inevitable business, namely realize the monitoring and early warning to different business; Also set up matrix model to the different field of same business, find out the certain event that analog value appears in different field, then coordinate alteration ruler, the data reached violating certain event carry out monitoring and early warning; To single business carry out with second, point, time, day, week, month, year for periods dimension, set up discrete fourier model, judge service period, look-ahead and monitoring and early warning are carried out to the data of single business; Namely threshold value of warning being set, when a threshold is exceeded, then carrying out early warning, realize the monitoring to business datum, so, by arranging early warning threshold values accurately, can monitor more accurately business datum.
In technique scheme, preferably, by described middleware, the interface mode that described business datum is sent to described supervisory system is comprised: File mode and/or Http mode.
In this technical scheme, supervisory system includes but not limited to File mode and/or Http mode for the interface mode that middleware system provides, namely File mode can be passed through: specified file path, filename expression formula (comprising Business Name, generation time, GUID unique identifying number), and Http mode: transmission XML data, business datum is entered in supervisory system by named variable title (comprising Business Name, generation time, GUID unique identifying number).
By above technical scheme, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast.
Accompanying drawing explanation
Fig. 1 shows according to an embodiment of the invention based on the process flow diagram of the business datum method for supervising of middleware;
Fig. 2 shows according to an embodiment of the invention based on the block diagram of the business datum supervisory system of middleware;
Fig. 3 shows the concrete schematic diagram of the business datum method for supervising based on middleware according to an embodiment of the invention;
Fig. 4 shows the schematic diagram of the Cassandra cluster setting up load balancing;
Fig. 5 shows just business datum and is deposited into the schematic diagram stored in rule in NoSQL database Cassandra;
Fig. 6 shows the method schematic diagram all XML data be deposited into the form of value in NoSQL database Cassandra.
Embodiment
In order to more clearly understand above-mentioned purpose of the present invention, feature and advantage, below in conjunction with the drawings and specific embodiments, the present invention is further described in detail.It should be noted that, when not conflicting, the feature in the embodiment of the application and embodiment can combine mutually.
Set forth a lot of detail in the following description so that fully understand the present invention; but; the present invention can also adopt other to be different from other modes described here and implement, and therefore, protection scope of the present invention is not by the restriction of following public specific embodiment.
Fig. 1 shows according to an embodiment of the invention based on the process flow diagram of the business datum method for supervising of middleware.
As shown in Figure 1, according to the business datum method for supervising based on middleware of the present invention, comprising: step 102, the business datum from different application systems is sent to supervisory system by middleware, to monitor described business datum; Step 104, carries out data storage and data modeling according to default storage rule to described business datum; Step 106, carries out Data classification to the described business datum through described data storage and described data modeling process; Step 108, sets up service correlation model according to described Data classification, to carry out track following to business datum described in every bar; Step 110, according to described service correlation model, arranges threshold value of warning, to monitor described business datum.
In this technical scheme, various different application system is connected by middleware, the business datum of different application systems is sent to supervisory system, and stored by data, data modeling, and can the ability of not designated data structure in advance in conjunction with NoSQL database, the structure of data model is designed to dynamic change, oneself can increase in actual O&M process to make model, self-teaching, then Data classification is passed through, set up service correlation model, monitoring and early warning threshold value is set, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast, learn by using large data teaching display stand, and pass through intelligent modeling, intelligent classification, intelligent predicting early warning reaches the target of monitoring business data.
In technique scheme, preferably, before described business datum is sent to described supervisory system by described middleware, described business datum is judged whether to be sent to described supervisory system, and when judged result is for being, after described business datum being converted to XML data by adapter, be sent to described supervisory system; And according to described default storage rule, based on NoSQL database, the storage of described data and described data modeling are carried out to described business datum.
In this technical scheme, before business datum is sent to supervisory system by middleware, that is, by business datum access or can select when being sent to corresponding application system to monitor the need of to this business datum, and when needs are monitored, by adapter, business datum is converted to XML (Extensible Markup Language, markup language) data, to realize the mark to needing the business datum monitored, and based on NoSQL (Not Only StructuredQuery Language, being not only Structured Query Language (SQL)) database Cassandra (distributed NoSQL Database Systems of increasing income) carries out data storage and modeling, so that realize the storage to business datum, modeling, classification, trajectory track etc., and then the monitoring realized business datum.
In technique scheme, preferably, according to described default storage rule, described XML data is stored in described NoSQL database Cassandra, and carries out described data modeling; Calculate the weighted value of the field of XML data described in every bar, and according to described weighted value, described Data classification is carried out to described XML data.
In this technical scheme, by default storage rule, XML data is stored in NoSQL database, to realize the process to super amount data, and carries out data modeling, the data of storage can be made to have standardization, be convenient to follow-uply search and/or monitor; By calculating the weighted value of data field, intelligent classification is carried out to data, for realizing providing favourable prerequisite guarantee to the monitoring of business datum.
In technique scheme, preferably, according to described service correlation model, the probability matrix that the business that is associated occurs, to determine the inevitable business in described associated services, the threshold value of warning of other business in associated services according to described inevitable business setting, to monitor described business datum.
In this technical scheme, according to service correlation model, list the probability matrix that all related services occur, thus find the inevitable business in related service, and monitoring and early warning is carried out to the data violating inevitable business, namely realize the monitoring and early warning to different business; Also set up matrix model to the different field of same business, find out the certain event that analog value appears in different field, then coordinate alteration ruler, the data reached violating certain event carry out monitoring and early warning; To single business carry out with second, point, time, day, week, month, year for periods dimension, set up discrete fourier model, judge service period, look-ahead and monitoring and early warning are carried out to the data of single business; Namely threshold value of warning being set, when a threshold is exceeded, then carrying out early warning, realize the monitoring to business datum, so, by arranging early warning threshold values accurately, can monitor more accurately business datum.
In technique scheme, preferably, by described middleware, the interface mode that described business datum is sent to described supervisory system is comprised: File mode and/or Http mode.
In this technical scheme, supervisory system includes but not limited to File mode and/or Http mode for the interface mode that middleware system provides, namely File mode can be passed through: specified file path, filename expression formula (comprising Business Name, generation time, GUID unique identifying number), and Http mode: transmission XML data, business datum is entered in supervisory system by named variable title (comprising Business Name, generation time, GUID unique identifying number).
Fig. 2 shows according to an embodiment of the invention based on the block diagram of the business datum supervisory system of middleware.
As shown in Figure 2, according to the business datum supervisory system 200 based on middleware of the present invention, comprising: data conversion module 202, for the business datum from different application systems is sent to supervisory system by middleware, to monitor described business datum; Data memory module 204, for carrying out data storage and data modeling according to default storage rule to described business datum; Data categorization module 206, for carrying out Data classification to the described business datum through described data storage and described data modeling process; Business model module 208, for setting up service correlation model according to described Data classification, to carry out track following to business datum described in every bar; Data monitoring module 210, for according to described service correlation model, arranges threshold value of warning, to monitor described business datum.
In this technical scheme, various different application system is connected by middleware, the business datum of different application systems is sent to supervisory system, and stored by data, data modeling, and can the ability of not designated data structure in advance in conjunction with NoSQL database, the structure of data model is designed to dynamic change, oneself can increase in actual O&M process to make model, self-teaching, then Data classification is passed through, set up service correlation model, monitoring and early warning threshold value is set, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast, learn by using large data teaching display stand, and pass through intelligent modeling, intelligent classification, intelligent predicting early warning reaches the target of monitoring business data.
In technique scheme, preferably, also comprise: judge module 212, for before described business datum is sent to described supervisory system by described middleware, judge whether described business datum to be sent to described supervisory system; Described data conversion module 202 specifically for: when judged result is for being, after described business datum being converted to XML data by adapter, be sent to described supervisory system; And described data memory module specifically for: according to described default storage rule to described business datum based on NoSQL database carry out described data store and described data modeling.
In this technical scheme, before business datum is sent to supervisory system by middleware, that is, by business datum access or can select when being sent to corresponding application system to monitor the need of to this business datum, and when needs are monitored, by adapter, business datum is converted to XML (Extensible Markup Language, markup language) data, to realize the mark to needing the business datum monitored, and based on NoSQL (Not Only StructuredQuery Language, being not only Structured Query Language (SQL)) database Cassandra (distributed NoSQL Database Systems of increasing income) carries out data storage and modeling, so that realize the storage to business datum, modeling, classification, trajectory track etc., and then the monitoring realized business datum.
In technique scheme, preferably, described data memory module 204 specifically for: according to described default storage rule, described XML data is stored in described NoSQL database Cassandra, and carries out described data modeling; And described data categorization module 206 is specifically for the weighted value calculating the field of XML data described in every bar, and according to described weighted value, described Data classification is carried out to described XML data.
In this technical scheme, by default storage rule, XML data is stored in NoSQL database, to realize the process to super amount data, and carries out data modeling, the data of storage can be made to have standardization, be convenient to follow-uply search and/or monitor; By calculating the weighted value of data field, intelligent classification is carried out to data, for realizing providing favourable prerequisite guarantee to the monitoring of business datum.
In technique scheme, preferably, described data monitoring module 210 specifically for: according to described service correlation model, the probability matrix that the business that is associated occurs, to determine the inevitable business in described associated services, the threshold value of warning of other business in associated services according to described inevitable business setting, to monitor described business datum.
In this technical scheme, according to service correlation model, list the probability matrix that all related services occur, thus find the inevitable business in related service, and monitoring and early warning is carried out to the data violating inevitable business, namely realize the monitoring and early warning to different business; Also set up matrix model to the different field of same business, find out the certain event that analog value appears in different field, then coordinate alteration ruler, the data reached violating certain event carry out monitoring and early warning; To single business carry out with second, point, time, day, week, month, year for periods dimension, set up discrete fourier model, judge service period, look-ahead and monitoring and early warning are carried out to the data of single business; Namely threshold value of warning being set, when a threshold is exceeded, then carrying out early warning, realize the monitoring to business datum, so, by arranging early warning threshold values accurately, can monitor more accurately business datum.
In technique scheme, preferably, by described middleware, the interface mode that described business datum is sent to described supervisory system is comprised: File mode and/or Http mode.
In this technical scheme, supervisory system includes but not limited to File mode and/or Http mode for the interface mode that middleware system provides, namely File mode can be passed through: specified file path, filename expression formula (comprising Business Name, generation time, GUID unique identifying number), and Http mode: transmission XML data, business datum is entered in supervisory system by named variable title (comprising Business Name, generation time, GUID unique identifying number).
Fig. 3 shows the business datum method for supervising step schematic diagram based on middleware according to an embodiment of the invention.
As shown in Figure 3, the business datum method for supervising step based on middleware according to an embodiment of the invention, comprising:
Step 1, middleware connects various different application system, the data (all business datums of different application systems) of dissimilar form are converted to XML data by adapter, user only needs unlabeled data the need of being deposited in monitor database, namely can monitor data.
Step 2, system can pass through rational data storage method, not restricted document structure, and data is stored by corresponding storage rule.Calculate the similarity between each XML data document by the cosine law, carry out intelligent modeling to XML data document, set up documentation release storehouse.
Step 3, by the similarity of XML, comes to use TF-IDF (Term frequency-inverse document frequency, weighting technique) to calculate weighted value to the field of similar data, and then generates corresponding classifying rules.
Step 4, finds out the incidence relation between each business by the field that weighted value is larger, lists all business models found in system, is found the business track of wall scroll data, for business diagnosis provides authentic data by business model.
Step 5, according to associated services, lists the probability matrix that all dependent events occur, thus finds the certain event in related service, and carry out monitoring and early warning to the data violating certain event.Also set up matrix model to the different field of same business, find out the certain event that analog value appears in different field, then coordinate alteration ruler, the data reached violating certain event carry out early warning.To single business carry out with second, point, time, day, week, month, year for periods dimension, set up discrete fourier model, judge service period, look-ahead and early warning are carried out to the data of single business, reversal periods prediction is carried out to certain event.
Step 6, management intelligence generate out data model, disaggregated model, business model, predict across type service prediction, the prediction of traffic forecast of the same type, single service period, certain event reversal periods, and add alteration ruler in time.Divide three-level management to the monitor data having generated out, first order Surveillance center manages, and second level business responsible official managing subscribing rule, the third level manages respective monitor data by person liable.
Concrete methods of realizing:
One, be sent in supervisory system by the business datum of different application systems by middleware, its step comprises:
1) business datum intelligent monitoring provides the interface of File, Http two kinds of modes for middleware system.
2) middleware system is having in adapter class the interface realizing supervisory system.
3) mode of middleware system configuration monitoring system data typing: a) File mode: specified file path, filename expression formula (comprising Business Name, generation time, GUID unique identifying number); B) Http mode: transmission XML data, named variable title (comprising Business Name, generation time, GUID unique identifying number).
4) all application systems data access or can select when sending data to application system to monitor the need of by data.
Owing to employing middleware, be easy to the data of each system to associate, and business datum intelligent monitoring both can be considered as independently system, also can be considered as an assembly module of middleware.Operationally, all functions of business datum intelligent monitoring itself are also the parts being deployed in middleware, can distributedly dispose.The middleware product Bull ESB that the present invention is emerging in can using, the data inputting module used in invention is the configuration feature of middleware oneself itself, does not therefore do too much explanation.
Two, data storage and data modeling are carried out to business datum
The present invention is in the basis of large database concept Cassandra, stores and intelligent modeling data.Specific implementation step:
1) the Cassandra cluster of load balancing is set up, as shown in Figure 4.
2) by the Column structure of Cassandra standard, the monitor data accessed into is at first deposited in NoSQL database, stored in rule as shown in Figure 5:
Set up Key Space:ESB Cloud, Column Family:Message Data.
Business datum is combined into the key value of Column with Business and date created and GUID, all data is deposited in Message Data.Set up following Columns:
Wherein using the field name in XML structure and attribute-name all as Column Name, respective value is as the value of Column.After such operation, the content of origination message can be known not only, and be easy to find the value of primary fields in origination message.In this structure, last value is only recorded to Repeating Field.Because if field repeats, then corresponding field is worth and declines, if need detailed value, and can by finding in body.Wherein XMLFiled_Filed* is complete trails, comprises its parent information.
3) all XML data are deposited in NoSQL database with the form of value, method as shown in Figure 6:
Set up Column Family:Business Data.
Using field value as Key, be combined into Business and date created and GUID as Column Name, corresponding field is as Value.Set up following Columns:
After such operation, all business with value can be together in series, because non-relation data is not to the restriction of Column, make the expansion of different business very convenient.Key value again owing to by Business and date created and GUID combination being Message Data itself, thus with regard to be easy to find the content in origination message be after data intelligence monitoring provide many help.
Three, according to the TF-IDF weighted value of calculated field, business datum is classified
Show at Message Data, have recorded the information of every bar data, now for data set up data characteristics, data characteristics comprises: set of data structures, Data classification collection, data record set.First suppose that the set of data structures of every bar record is not identical.Then can set up the feature list of structure collection, suppose there is S representative structure collection, represent Column Name (only comprising XML Field part, is also with rule below) with F, then there is a structure collection in every bar record, is listed as follows:
S(1_1)＝[F1，F2，F3，F4....]
S (1_2)=[F1, F2, F3, F4....]-----is (with S
1structure collection is identical)
S (1_3)=[F1, F2, F3, F4....]-----is (with S
1structure collection is identical)
S (2)=[F2, F6, F8 ... ..]-----(there is new structure)
S(3)＝[...]
....
Set up the total collection S of set of data structures
0, comprise all Column Name and gather, and result set is removed identical structure collection data:
S (0)=[F1, F2, F3, F4, F5, F6 ... .] comprise all Column Name.
S (1)=[F1, F2, F3, F4 ... .] remove the result after repetition.
S(2)＝[F2，F6，F8…..]
S(3)＝[…]
....
The word frequency TF (i) of each Column Name of present calculating, supposes that freq (i) is the frequency that Fi occurs in all data structures, and making Other Columns (i) represent is S
0in the set of other Column Name.Maximum frequency maxOthers (i) is then:
max(freq(z),z∈OtherColumns(i)
Finally, calculate TF (i), be then:
Due to some field probability of occurrence more greatly but few of value, use anti-document frequency in this way, if N is the total set of data structures number after duplicate removal, n (i) is for being expressed as the quantity of the set of data structures that ColumnName i occurs in all data structures.Then anti-document frequency is:
Then the weighted value of each field is:
TF_IDF(i)＝TF(i)×IDF(i)
Because the Column Name of individual data can not be too many, generally within 100, be also often greater than 5, therefore suppose the field that weighted value is important or major key or combine major key, all have such method here.The first five more important for each set of data structures weight ratio ColumnName is formed a new set, the short covering collection less than 5.Then there is following list:
S(1)＝[F1，F4,F7，F9，F10]
S(2)＝[F8，F14，F17，F19，F110]
...
S (n)=[F1, F4, F7, F9, F10] finds the same with the structure of S (1).
...
So the data set that we are more all, discovery is the same, then think identical structure, and just version information is different.The data structure of whole platform can both automatic classification in this way, and recording version information.And by above weight information, set of data structures information, set of data structures version information, data structure classified information is all deposited in NoSQL database, and calculates the data structure classification value of every bar Message Data.
After set of data structures branches away, we just can start the Data classification set calculating all data.Still use the data in Message Data.Take out a data textural classification from data structure class table, and then in Message Data, find all data that data structure classification value is identical, carry out intelligent data modeling.S (i) represents the data textural classification that we get.R represents R (1) in all data acquisitions that we get until R (N), N are record count.Column Name all in S (i) forms a new list, adds up to J.
S(i)＝[F(1)，F(2)，F(3)，F(4)，F(5)....F(j)]
Calculate data set F (j) of the value value below each field=[V (j, 1), V (j, 2) ... V (j, N)], v (j, n) indicates the value of jth corresponding to Column Name of the n-th data.After removing repeating data, can think that often organizing Column Name finds out all event models, remove after single incident model and event model number be greater than the Column Name of more than event model threshold values (acquiescence 10000), the just remaining Data classification collection that we will look for.Then arrange according to the weight of each field, namely can occur the intelligent classification data set that we wish to obtain.
Four, service correlation model is set up according to Data classification, to carry out track following
The data of Business table are used when business model.In previous step, we have known the weighted value of each field in the type of data structure and data structure, and we can set up service correlation model on this basis.In Business table, we are linked togather all serial datas of same value, in this table, find matrix table by the type of data structure.Suppose that the data structure monitored is N, data structure S (n) is its n-th data structure, data structure S (n) containing J ColumnName, with p (n, j, k) represent the j field whether containing ratio in data structure k of data structure n.The computation rule contained can find one of 5 critical fielies before k data in the data of Business Data, if the person of have found is for containing by j field value.Then:
Data structure n is represented, the set of relationship of j field and other all data structures with array p (n, j):
p(n,j)＝[p(n,j,1)p(n,j,2)...p(n,j,k)......p(n,j,K)]
The relation of data structure n and other data structure is represented with array p (n):
Therefore when need by the value of a random input go for we want the data of looking for time, first the data of all valuable data structure matched can be found out in Business Data, when the data by data structure go to find again, all valuable information in all p (n) will be listed.Such as, in education sector, when inquiring the student information of a student, business model can be automatically found the teacher information relevant to it, class's information, exam information, specialized information etc.
When getting up above business model and time correlation, we are just easy to each data to find its life track.Such as, in medical treatment, intelligent monitoring to the medical history of each patient, and can find relevant doctor, expense, medicine etc. information in each medical center, thus reaches track following.The display of these data is all well offer help for related service personnel carry out monitor data analysis.
Five, set up business rule prediction to predict with time cycle property, business datum is monitored
Business rule prediction is predicted can calculate utilizing the result of the 4th step with time cycle property.
1) Forecasting Methodology between different business is as follows: in four, we have known the incidence relation between business and business, each data structure represents a kind of business, the Sn that will monitor in monitor data structure occurs that the sample space of Sm is S (n, m), if the J contained in Sn Column, containing K field in Sm, with P (j, k) the k field probability that Sm has occurred when Sn occurs of the corresponding Sm of a jth Column is represented, the weighted value of j field in Sn structure is represented with PN (j), the weighted value of k field in Sm structure is represented with PM (k), S (n, m) predicted value is represented:
If represent that the sample space that business rule is predicted, N are all data structure numbers of S with S.With Smax, suppose in the data monitored, to occur a pair inevitable probability event, then S (n, m) the probability size that Sm appears in Sn can be described, then Smax is at least also an inevitable probability event, if Smax is zero, then all events in expression system are all separate.We use Smax as sample canonical.
Represent with S (n, m)/Smax the change that Sn occurs, the probable value PNM (n, m) that corresponding Sm occurs, namely occur as business n and go to predict the probable value of m appearance.
Calculate all PNM values in system, and early warning threshold values f is set, be assumed to be 0.9.If PNM value is greater than f, is set to necessary monitored object, even occurs N data then, then must have M data, otherwise report to the police.
2) single business datum Forecasting Methodology: in step 4, found the field can classified in all data structures, and all classification values of sorting field.Whether correctly the interrelated relation of different field value is utilized now to carry out the data of checklist business.Such as, if the Zhi Shi Guangdong Province of economizing in address information, and the value in the city of address is Beijing, and this is obviously have problem.This Forecasting Methodology can address this is that.Each sorting field represents a kind of business, and sorting field is less than 1 not analyzing.The value sample space that the corresponding sorting field Fm of value of sorting field Fn occurs is F (n, m), if the J contained in Fn classification value, containing K classification value in Fm, with P (j, k) the k field probability that Fm has occurred when Fn occurs of the corresponding Fm of a jth classification value is represented, the value with matrix representation F (n, m):
Calculate the value V (n, m) of F (n, m) as early warning in addition again, represent the weighted value of n sorting field n with PN (n), represent the maximal value of all sorting fields product each other with MaxPNN, then predictive value is:
Predicted value FV is represented with F (n, m) and the product of V (n, m):
FV(n,m)＝F(n,m)*V(n,m)
Calculate all FV values in system, and early warning threshold values f is set, be assumed to be 0.9.If FV value is greater than f, being set to must monitored object, and even P (j, k) * V (n, m) is greater than 0.9, and j appears in sorting field N and j data should appear in sorting field M, otherwise reports to the police.Add modified values simultaneously, report to the police if find meaningless, then Lookup protocol predictive value V (n, m)=0.
3) single service period prediction: suppose that all business have periodically, only the cycle was formed by stacking by multiple cycle.Following Main Analysis goes out some valuable periodic traffic.Suppose all valuable periodic traffic by by second, point, time, multiple cycles in units of day, week, month, year sexual behavior part form, and with second, point, time, day, week, month, year maximum event numbers be respectively (1,60,24,31,5,12,1).So each business is made up of 1+60+24+31+5+12+1=134 recurrent event, and the value of each periodic function only has 1 and-1, namely betides anti-generation.
f(t)＝K
1f
1(t)+K
2f
2(t)+K
3f
3(t)+...+...+K
134f
134(t)
Wherein K
ifor the coefficient that each periodic function occurs, and K
ivalue be 0 or 1.Such as one business only having the week just to occur, can with an event diurnal periodicity (generation event) and two cycle events (anti-event occurs) superposition.
After having had above rule, can start to have how many seconds, point, time, day, week, month, year periodic function.First in supposing the system, the data of this business are all normal data and meet compounding period function, by the interval calculation of all adjacent events out (t1, t2, t3, t4, t5....tn) suppose there is n+1 bar record.Finally all time all uses
Tn=a
1year+a
2the moon+a
3week+a
4day+a
5time+a
6divide+a
7second
Meet Hour Minute Second on Sunday on days respectively with maximum greedy algorithm, calculate a
1, a
2, a
3, a
4, a
5, a
6, a
7.Record a
2, a
3, a
4, a
5, a
6.Be counted as all record a
2, a
3, a
4, a
5, a
6composition number, and record often group there is identical Probability p (n), if p (n) is close to 0, abandons this group result, when supposing p (n) <0.01 here, abandon result.Form by how many and just prove have how many periodic events to exist.By new a
2, a
3, a
4, a
5, a
6combination, reuse maximum greedy algorithm calculated respectively how many, point, time, day, week, month, year periodic function.
4) the reversal periods prediction of certain event: when supposing that in five, business A occurs, business B must occur, then, after we can count all business B generation, being distributed as of the time interval of business A should be able to occur:
Namely arrive at most in mathematical expectation, how many business should be occur within the covariance of mathematical expectation.Exceed reservation value when the time, then carry out early warning.
Six, monitoring management and subscription management is carried out
Management intelligence generate out data model, disaggregated model, business model, predict across type service prediction, the prediction of traffic forecast of the same type, single service period, certain event reversal periods, and add alteration ruler in time.Divide three-level management to the monitor data having generated out, first order Surveillance center manages, and second level business responsible official managing subscribing rule, the third level manages respective monitor data by person liable.
More than be described with reference to the accompanying drawings technical scheme of the present invention, the intelligent monitoring to different business data can be realized, effectively improve versatility and the adaptability to the change of business datum and the change of business rule, and the monitoring demand of large conglomerate can be met fast.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1., based on a business datum method for supervising for middleware, it is characterized in that, comprising:
Business datum from different application systems is sent to supervisory system by middleware, to monitor described business datum;
According to default storage rule, data storage and data modeling are carried out to described business datum;
Data classification is carried out to the described business datum through described data storage and described data modeling process;
Service correlation model is set up, to carry out track following to business datum described in every bar according to described Data classification;
According to described service correlation model, threshold value of warning is set, to monitor described business datum.
2. the business datum method for supervising based on middleware according to claim 1, it is characterized in that, before described business datum is sent to described supervisory system by described middleware, described business datum is judged whether to be sent to described supervisory system, and when judged result is for being, after described business datum being converted to XML data by adapter, be sent to described supervisory system; And
Carry out described data to described business datum based on NoSQL database according to described default storage rule to store and described data modeling.
3. the business datum method for supervising based on middleware according to claim 2, is characterized in that, described XML data is stored in described NoSQL database Cassandra, and carries out described data modeling according to described default storage rule;
Calculate the weighted value of the field of XML data described in every bar, and according to described weighted value, described Data classification is carried out to described XML data.
4. the business datum method for supervising based on middleware according to claim 3, it is characterized in that, according to described service correlation model, the probability matrix that the business that is associated occurs, to determine the inevitable business in described associated services, the threshold value of warning of other business in associated services according to described inevitable business setting, to monitor described business datum.
5. the business datum method for supervising based on middleware according to any one of claim 1 to 4, be is characterized in that, comprised by the interface mode that described business datum is sent to described supervisory system by described middleware: File mode and/or Http mode.
6., based on a business datum supervisory system for middleware, it is characterized in that, comprising:
Data conversion module, for the business datum from different application systems is sent to supervisory system by middleware, to monitor described business datum;
Data memory module, for carrying out data storage and data modeling according to default storage rule to described business datum;
Data categorization module, for carrying out Data classification to the described business datum through described data storage and described data modeling process;
Business model module, for setting up service correlation model according to described Data classification, to carry out track following to business datum described in every bar;
Data monitoring module, for according to described service correlation model, arranges threshold value of warning, to monitor described business datum.
7. the business datum supervisory system based on middleware according to claim 6, it is characterized in that, also comprise: judge module, for before described business datum is sent to described supervisory system by described middleware, judge whether described business datum to be sent to described supervisory system;
Described data conversion module specifically for: when judged result is for being, after described business datum being converted to XML data by adapter, be sent to described supervisory system; And
Described data memory module specifically for: according to described default storage rule to described business datum based on NoSQL database carry out described data store and described data modeling.
8. the business datum supervisory system based on middleware according to claim 7, it is characterized in that, described data memory module specifically for: according to described default storage rule, described XML data is stored in described NoSQL database Cassandra, and carries out described data modeling; And
Described data categorization module specifically for the weighted value calculating the field of XML data described in every bar, and carries out described Data classification according to described weighted value to described XML data.
9. the business datum supervisory system based on middleware according to claim 8, it is characterized in that, described data monitoring module specifically for: according to described service correlation model, the probability matrix that the business that is associated occurs, to determine the inevitable business in described associated services, the threshold value of warning of other business in associated services according to described inevitable business setting, to monitor described business datum.
10. the business datum supervisory system based on middleware according to any one of claim 6 to 9, be is characterized in that, comprised by the interface mode that described business datum is sent to described supervisory system by described middleware: File mode and/or Http mode.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510505642.1A CN105005525B (en) | 2015-08-17 | 2015-08-17 | Business datum monitoring method and system based on middleware |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510505642.1A CN105005525B (en) | 2015-08-17 | 2015-08-17 | Business datum monitoring method and system based on middleware |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105005525A true CN105005525A (en) | 2015-10-28 |
| CN105005525B CN105005525B (en) | 2018-02-02 |
Family
ID=54378204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510505642.1A Active CN105005525B (en) | 2015-08-17 | 2015-08-17 | Business datum monitoring method and system based on middleware |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105005525B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105550314A (en) * | 2015-12-14 | 2016-05-04 | Tcl集团股份有限公司 | Data statistics method, device and system based on combined major key |
| CN109215765A (en) * | 2018-09-20 | 2019-01-15 | 武汉谷泰软件科技有限公司 | The information for hospital integrated platform system of heterogeneous service data interconnection intercommunication can be achieved |
| CN110619504A (en) * | 2018-06-19 | 2019-12-27 | 阿里巴巴集团控股有限公司 | Service change processing method, device and system and electronic equipment |
| CN110618911A (en) * | 2019-08-15 | 2019-12-27 | 中国平安财产保险股份有限公司 | Data monitoring method and device, storage medium and server |
| CN111143163A (en) * | 2019-12-13 | 2020-05-12 | 上海硬通网络科技有限公司 | Data monitoring method and device, computer equipment and storage medium |
| CN112422638A (en) * | 2020-10-28 | 2021-02-26 | 北京北明数科信息技术有限公司 | Data real-time stream processing method, system, computer device and storage medium |
| CN112906012A (en) * | 2019-12-03 | 2021-06-04 | 株洲中车时代电气股份有限公司 | Processing method and device for data with different structures and electronic equipment |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947853A (en) * | 2019-03-25 | 2019-06-28 | 阿里巴巴集团控股有限公司 | A kind of business activity change cognitive method, system and server |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1482566A (en) * | 2003-01-22 | 2004-03-17 | 余育新 | Distributed work-flow management platform |
| US20080034232A1 (en) * | 2006-08-03 | 2008-02-07 | Dell Products, Lp | System and method of managing heat in multiple central processing units |
| CN101197876A (en) * | 2006-12-06 | 2008-06-11 | 中兴通讯股份有限公司 | Method and system for multi-dimensional analysis of message service data |
-
2015
- 2015-08-17 CN CN201510505642.1A patent/CN105005525B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1482566A (en) * | 2003-01-22 | 2004-03-17 | 余育新 | Distributed work-flow management platform |
| US20080034232A1 (en) * | 2006-08-03 | 2008-02-07 | Dell Products, Lp | System and method of managing heat in multiple central processing units |
| CN101197876A (en) * | 2006-12-06 | 2008-06-11 | 中兴通讯股份有限公司 | Method and system for multi-dimensional analysis of message service data |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105550314A (en) * | 2015-12-14 | 2016-05-04 | Tcl集团股份有限公司 | Data statistics method, device and system based on combined major key |
| CN110619504A (en) * | 2018-06-19 | 2019-12-27 | 阿里巴巴集团控股有限公司 | Service change processing method, device and system and electronic equipment |
| CN110619504B (en) * | 2018-06-19 | 2023-05-02 | 阿里巴巴集团控股有限公司 | Service change processing method, device and system and electronic equipment |
| CN109215765A (en) * | 2018-09-20 | 2019-01-15 | 武汉谷泰软件科技有限公司 | The information for hospital integrated platform system of heterogeneous service data interconnection intercommunication can be achieved |
| CN110618911A (en) * | 2019-08-15 | 2019-12-27 | 中国平安财产保险股份有限公司 | Data monitoring method and device, storage medium and server |
| CN110618911B (en) * | 2019-08-15 | 2022-12-09 | 中国平安财产保险股份有限公司 | Data monitoring method and device, storage medium and server |
| CN112906012A (en) * | 2019-12-03 | 2021-06-04 | 株洲中车时代电气股份有限公司 | Processing method and device for data with different structures and electronic equipment |
| CN111143163A (en) * | 2019-12-13 | 2020-05-12 | 上海硬通网络科技有限公司 | Data monitoring method and device, computer equipment and storage medium |
| CN111143163B (en) * | 2019-12-13 | 2024-04-16 | 上海硬通网络科技有限公司 | Data monitoring method, device, computer equipment and storage medium |
| CN112422638A (en) * | 2020-10-28 | 2021-02-26 | 北京北明数科信息技术有限公司 | Data real-time stream processing method, system, computer device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105005525B (en) | 2018-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105005525A (en) | Middleware based service data monitoring method and system | |
| CN107729413B (en) | Regional traffic intelligent management system based on big data | |
| CN110188979B (en) | Water industry emergency decision generation method and device | |
| Fox | The role of ontologies in publishing and analyzing city indicators | |
| Baldominos et al. | DataCare: big data analytics solution for intelligent healthcare management | |
| CN111160867A (en) | Large-scale regional parking lot big data analysis system | |
| CN109754177A (en) | Pollution sources portrait label system, the building method of pollution sources portrait and its application | |
| Li et al. | Entity-relationship diagram | |
| CN110990479A (en) | City big data visualization system | |
| CN112527886A (en) | Data warehouse system based on urban brain | |
| Fadhel et al. | A comparison of time series databases for storing water quality data | |
| CN116383198A (en) | Decision analysis method and system based on big data | |
| Fox | The semantics of populations: A city indicator perspective | |
| Anastasiou et al. | Admsv2: A modern architecture for transportation data management and analysis | |
| Qazi et al. | Associative search through formal concept analysis in criminal intelligence analysis | |
| Radermacher et al. | Trends in environmental information processing | |
| Russell et al. | MPACT family trees: Quantifying academic genealogy in library and information science | |
| CN115718776A (en) | Big data application platform system | |
| Laurini et al. | Regional knowledge management and sustainable regional development: in quest of a research and knowledge agenda | |
| Billiet et al. | Evaluating possibilistic valid-time queries | |
| Liang | Allocation of multi-dimensional distance learning resource based on MOOC data | |
| Gochman et al. | Militarized disputes, incidents, and crises: Identification and classification | |
| Wang et al. | Research on the Construction and Application of Burma-Vietnam's Political Event Data Set | |
| Hudec et al. | Fuzzy data in relational databases | |
| RU2670842C1 (en) | Device of universal data storage and method of its formation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |