CN103152189B - A kind of collecting method based on business association and system - Google Patents
A kind of collecting method based on business association and system Download PDFInfo
- Publication number
- CN103152189B CN103152189B CN201110401963.9A CN201110401963A CN103152189B CN 103152189 B CN103152189 B CN 103152189B CN 201110401963 A CN201110401963 A CN 201110401963A CN 103152189 B CN103152189 B CN 103152189B
- Authority
- CN
- China
- Prior art keywords
- data
- gatherer process
- data item
- business
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000000034 method Methods 0.000 title claims abstract description 237
- 238000004458 analytical method Methods 0.000 claims abstract description 16
- 230000032683 aging Effects 0.000 claims abstract description 10
- 239000008186 active pharmaceutical agent Substances 0.000 claims description 67
- 238000013480 data collection Methods 0.000 claims description 15
- 238000012546 transfer Methods 0.000 claims description 15
- 238000003780 insertion Methods 0.000 claims description 11
- 230000037431 insertion Effects 0.000 claims description 6
- 238000007405 data analysis Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 230000002452 interceptive effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present invention relates to a kind of collecting method based on business association and system, data gathering process is associated for providing data supporting for more complicated network management traffic analysis by the method with business, described method is: for generating the step of data item queue, the data item multiple to be collected belonging to same business is placed by this step continuously, and the total data item to be collected belonging to different business is placed between two kinds of business data item corresponding respectively; For adopting some collecting thread groups to start the every of data gathering process order data dispatching item queue placement, complete the step of the data gathering of business association. The invention solves the nonsynchronous problem of business association data gathering, namely in setting-up time section, complete all data item of capturing service association, improve the ageing and accuracy that business datum is analyzed, simultaneously by the acquisition time discretize of uncorrelated for business data, reduce the concurrent pressure of data gathering.
Description
Technical field
The present invention relates to IT technical field, in particular to a kind of collecting method based on business association and system.
Background technology
Along with the development of network management technology, its management area, except the management of the network elements such as traditional communication, calculating and storage, also comprises the management of bearer service on equipment. In network management system, business analysis is one of main function of service management, and the Data Source that business is analyzed is in network management system to the collection of each equipment or system data, and present business analysis is more and more complicated. All data are carried out wheel and ask collection by traditional network management system collecting method, and the data acquisition time span required for the analysis of same business can be bigger, data analysis can not be processed in time, cause data analysis poor real; And, the data first gathered also may lose efficacy, and causes business analytical results and actual has deviation. Along with the development of network management system, it is necessary to new collection method meets more complicated business and analyzes demand.
China No. 200410034524 acquisition system and the method thereof that patent discloses configuration data in a kind of network management, in this approach, by the collection different according to different network element configuration with arrange different refreshing templates and template parser, to solve the extendability that net element information gathers. China No. 200910173983 the data parallel acquisition merging method and the system that patent discloses Simple Network Management Protocol equipment, this system comprises parallel control entity and one or more parallel acquisition entity, parallel control entity completes the management to parallel acquisition entity and unifies merger process with the performance data returned by each parallel acquisition entity, parallel acquisition entity in a parallel fashion from image data each collection point, to improve the operability of collecting efficiency and system.No. 200810117931st, China patent discloses a kind of collecting method and data acquisition equipment, the Information encapsulation of all collected network elements is become acquisition tasks to put into a task queue by described method, and start network element collecting thread pond, by the data of each network element in task queue described in the monitoring port processing of thread pool, to improve the efficiency of data gathering.
By to prior art and patent analyses, existing disclosed technology is to improve the efficiency of network management data gathering, extensibility as target, but, disclosed method does not all relate to data gathering and business association or the collection method based on business, this will cause the ageing of network management traffic data analysis and accuracy poor, and can not provide data supporting for more complicated network management traffic analysis. Therefore, this patent is by openly a kind of collecting method based on business and system, and the technical scheme of the present invention can improve the ageing of network management traffic data analysis and accuracy, for more complicated network management traffic analysis provides data supporting.
Summary of the invention
It is an object of the invention to, for overcoming prior art to improve the efficiency of network management data gathering, extensibility as target, but data gathering and business association or the collection method based on business is not all related to, thus cause the ageing of network management traffic data analysis and accuracy poor, and can not be the problem of more complicated network management traffic analysis offer data supporting, thus provide a kind of collecting method based on business association and system.
The technical problem to be solved in the present invention is: under the prerequisite of number of times reducing data gathering, it is to increase the real-time of network management traffic data analysis and accuracy, represent gatherer process queue with DS queue in following technical scheme.
For solving the problems of the technologies described above, the present invention proposes a kind of collecting method based on business association, and data gathering process is associated for providing data supporting for more complicated network management traffic analysis by the method with business, and described method is:
Step 1, for generating the step of data item queue, the data item multiple to be collected belonging to same business is placed by this step continuously, and the total data item to be collected belonging to different business is placed between two kinds of business data item corresponding respectively;
Step 2, for adopting some groups of threads to start the every of data gathering process order data dispatching item queue placement, completes the step of the data gathering of business association.
Wherein, described data item is collected smallest standalone unit.
In technique scheme, described step 1 comprises following steps further:
Multiple gatherer processes identical for collection period by each data collection service corresponding gatherer process respectively, and are divided in same DS queue by step 1-1;
Step 1-2, the multiple gatherer process each DS queue comprised carries out gatherer process sequence according to the total relation of the data item between gatherer process;
Step 1-3, carries out data item sequence in this gatherer process by each gatherer process according to the total relation of the data item between this gatherer process and close gatherer process.
Wherein, each DS queue described corresponding collecting thread group respectively.
Optionally, described gatherer process all adopts following four-tuple to describe:
DS=(S, T, Td, D);
Wherein, S represents business; T represents data collection cycle; Td represents the data acquisition time that can accept to postpone, and T >=Td;D represents the collection of data items that business packet contains, and D can not be empty, that is, data item associated by a business can not be empty, and in D, the attribute flags of the data item that different business has is " being re-used ".
Describing based on above-mentioned four-tuple, described step 1-1 judges that whether the T in the four-tuple that gatherer process is corresponding is identical, is divided in same DS queue if identical by gatherer process corresponding for two business;
Described step 1-2, by empty clearly for each DS queue; DS queue is inserted two gatherer process DS1、DS2, meet gatherer process DS1And DS2Compared with the data item that total data item is total with any two gatherer processes that this DS queue is corresponding at most; The next gatherer process DS of insertioni, meeting current DS queue top set place, length of window is Wi=q × TdiCollection of data items and gatherer process DSiTotal data item is maximum, and wherein, q represents the expectation of acquisition capacity; Repeat insertion gatherer process, until all gatherer processes are all inserted in corresponding DS queue;
Described step 1-3, by each corresponding for each gatherer process " being re-used " data item, is W at front and back window widthi=q × TdiIn scope, judge to have the position of the gatherer process of this data item with this gatherer process, if front, in this gatherer process, transfer forward the position of this data item, if rear, in this gatherer process, transfer the position of this data item backward, if existed rear front simultaneously, transfer the position of this data item to centre.
Optionally, the number of threads that described collecting thread group comprises adopts following formula to determine:
Wherein, M is the Thread Count of one group of collecting thread, and j is the business number that DS queue comprises, λiFor the ageing parameter that can arrange, being worth more little, the time of collection is more short, usually: 0 < λi≤ 1; T represents data collection cycle.
In technique scheme, described step 2 comprises following steps further:
Step 2-1, when the data gathering process started performs data gathering, first, checks whether this data item attribute is " being re-used ", if " being re-used ", enters next step; Otherwise carry out data gathering.
Step 2-2, will verify the Data Update time of this data item further, to determine whether this data item of Resurvey; If the Data Update time difference of system time and this data item is greater than the Td of gatherer process belonging to this data item, this collecting thread by this data item of Resurvey, otherwise, no longer gather.
Based on aforesaid method, the present invention provides a kind of data collecting system based on business association, and data gathering process is associated for providing data supporting for more complicated network management traffic analysis by this system with business, and described system is:
Data item queue generates subsystem, the data item multiple to be collected belonging to same business is placed by this subsystem continuously, the total data item to be collected belonging to different business is placed between two kinds of business data item corresponding respectively, exports the data item queue generated; With
Data gathering subsystem, this subsystem adopts some groups of threads to start the data item queue of data gathering process order data dispatching item queue generation subsystem output, completes the data gathering of business association.
In technique scheme, described data item queue generates subsystem and comprises further:
Multiple gatherer processes identical for collection period for each data collection service is distinguished a corresponding gatherer process, and are divided in same DS queue by DS queue generation module.With
DS queue order module, multiple gatherer process for each DS queue being comprised carries out gatherer process sequence according to the total relation of the data item between gatherer process, and then according to the total relation of the data item between this gatherer process and close gatherer process, each gatherer process is carried out data item sequence in this gatherer process.
Wherein, each DS queue described corresponding collecting thread group respectively.
Optionally, described gatherer process all adopts following four-tuple to describe:
DS=(S, T, Td, D);
Wherein, S represents business; T represents data collection cycle; Td represents the data acquisition time that can accept to postpone, and T >=Td; D represents the collection of data items that business packet contains, and D can not be empty, that is, data item associated by a business can not be empty, and in D, the attribute flags of the data item that different business has is " being re-used ";
Describing based on above four-tuple, also whether T for judging in the four-tuple that gatherer process is corresponding is identical for described DS queue generation module, is divided in same DS queue if identical by gatherer process corresponding for two business;
Described DS queue order module also for, by empty clearly for each DS queue; DS queue is inserted two gatherer process DS1、DS2, meet gatherer process DS1And DS2Compared with the data item that total data item is total with any two gatherer processes that this DS queue is corresponding at most; The next gatherer process DS of insertioni, meeting current DS queue top set place, length of window is Wi=q × TdiCollection of data items and gatherer process DSiTotal data item is maximum, and wherein, q represents the expectation of acquisition capacity; Repeat insertion gatherer process, until all gatherer processes are all inserted in corresponding DS queue;
Described DS queue order module also for, by each corresponding for each gatherer process " being re-used " data item, be all W at front and back window widthi=q × TdiIn scope, judge to have the position of the gatherer process of this data item with this gatherer process, if front, in this gatherer process, transfer forward the position of this data item, if rear, in this gatherer process, transfer the position of this data item backward, if existed rear front simultaneously, transfer the position of this data item to centre.
Optionally, described data gathering subsystem comprises further:
Item status inquiry and strategy generating module, for checking whether data item attribute is " being re-used ", if " being re-used ", confirm acquisition module by the input of this collection item; Otherwise carry out data gathering; With
Confirm acquisition module, for verifying the Data Update time of this data item further, to determine whether this data item of Resurvey; If the Data Update time difference of system time and this data item is greater than the Td of gatherer process belonging to this data item, this collecting thread by this data item of Resurvey, otherwise, no longer gather.
Optionally, described data gathering subsystem also comprises number of threads determination module, and this module determines, according to following formula, the number of threads that each collecting thread group comprises:
Wherein, M is the Thread Count of one group of collecting thread, and j is the business number that DS queue comprises, λiFor the ageing parameter that can arrange, being worth more little, the time of collection is more short, usually: 0 < λi≤ 1; T represents data collection cycle.
The useful effect of the present invention is:
1. the data item queue order technology that the present invention adopts, can ensure that all data item with business association complete to gather within the Td time, the data acquisition time span that a business can be avoided to be associated is excessive, causes part data failure, it is to increase the ageing and accuracy that business datum is analyzed.
2. for optimize acquisition performance further, the present invention adopts multiplexing discrimination technology, reduces the probability of " being re-used " data item Resurvey, reduces the number of times of data gathering.
3. the method is while concentrating service related data item acquisition time, reaches the effect of uncorrelated for business data item acquisition time discretize, thus reduces the concurrent pressure of data gathering.
Accompanying drawing explanation
Fig. 1-a is the schematic flow sheet of the collecting method based on business association of the present invention;
Fig. 1-b is method entity relation schematic diagram described in technical solution of the present invention;
Fig. 2 is collecting thread workflow diagram of the present invention;
Fig. 3 is embodiment of the present invention interactive television system simple relation figure;
Fig. 4 is described method entity graph of a relation taking interactive television system business datum analysis as example describes.
Embodiment
Below, the collecting method based on business association of the present invention is described in detail by business datum analysis by reference to the accompanying drawings and in interactive television system.
The present invention provides a kind of collecting method based on business association, it is characterised in that, the method adopts following 4 entities to realize: data item, gatherer process, gatherer process queue and collecting thread group, and specific implementation method main points are as follows:
1) gatherer process is corresponding with a business, and each gatherer process can comprise multiple data item, and each data item can be associated by multiple gatherer process again; A gatherer process queue comprises the identical gatherer process of multiple collection period, and a gatherer process can only belong to a queue; A collecting thread group associates one by one with a gatherer process queue;
2) collecting thread that collecting thread group comprises completes the collection of multiple data item that this business packet contains within collection time of lag that business is preset;
Optimizing, when belonging to two and above business when a data item, first collecting thread can judge the Data Update time of this data item, needs this data item of Resurvey to determine whether.
Wherein, according to the difference of collection period, gatherer process is put into different queues, that is, in each gatherer process queue, all gatherer processes have identical collection period, and each gatherer process queue is gathered by one group of collecting thread.
In order to realize above-mentioned purpose, the present invention also adopts a kind of four-tuple to describe each gatherer process, i.e. each gatherer process collected cycle and the acquisition time constraint accepted, specifically can describe by a four-tuple: DS=(S, T, Td, wherein, D), S represents business, T represents data collection cycle, and Td represents the data acquisition time that can accept to postpone, and T >=Td, D represents the collection of data items of associated services, D can not be empty, that is, data item associated by a business can not be sky; A data item can, associated by multiple business, when being associated with two and above business when a data item, will be " being re-used " for the setup of attribute of this data item.
Optionally, the number of threads of a collecting thread group depends on collection period and the Td of the data collection service that gatherer process is corresponding in DS queue, namelyWherein M is the Thread Count of one group of collecting thread, and j is the DS number that queue comprises, λiFor the ageing parameter that can arrange, being worth more little, the time of collection is more short, usually: 0 < λi≤ 1, but in systems in practice, actual number of threads can more than theoretical value.
Optimize, when collecting thread performs data gathering, can check whether this data item attribute is " being re-used ", if " being re-used ", the Data Update time of this data item will be verified further, to determine whether Resurvey;If the Data Update time difference of system time and this data item is greater than the Td of DS belonging to this data item, this collecting thread by this data item of Resurvey, otherwise, no longer gather.
As shown in Fig. 1-a, this figure is the schematic flow sheet of a kind of collecting method based on business association of the present invention, and described method is:
101) for generating the step of data item queue, the data item multiple to be collected belonging to same business is placed by this step continuously, the total data item to be collected belonging to different business is placed between two kinds of business data item corresponding respectively, and the implementation process of a kind of optional didactic generation data item queue is as follows:
Step 101-1, clear empty DS queue;
Step 101-2, inserts two gatherer process DS in DS queue1、DS2, meet gatherer process DS1And DS2Compared with the data item that total data item is total with any two gatherer processes that this DS queue is corresponding at most;
Step 101-3, the next gatherer process DS of insertioni, meeting current DS queue top set place, length of window is Wi=q × TdiCollection of data items and gatherer process DSiTotal data item is maximum, and wherein, q represents the expectation of acquisition capacity;
Step 101-4, repeating step 101-3, until all gatherer processes are all inserted in corresponding DS queue;
Each " being re-used " data item that step 101-5 is corresponding to each gatherer process is all W at front and back window widthi=q × TdiIn scope, judge to have the position of the gatherer process of this data item with this gatherer process, if front, in this gatherer process, transfer forward the position of this data item, if rear, in this gatherer process, transfer the position of this data item backward, if existed rear front simultaneously, transfer the position of this data item to centre.
Wherein, the implementation process of above-mentioned generation data item queue can also adopt the method based on dynamic programming: the method inserts a gatherer process every time in DS queue, and after meeting this gatherer process of insertion, in current DS queue, all gatherer processes window before and after it is all Wi=q × TdiScope in total data item sum maximum; Repeat insertion, until all gatherer processes are all inserted in DS queue; In gatherer process, data item location is transferred as above-mentioned steps 101-5. In addition, it is also possible to utilize the method such as artificial neural network, genetic algorithm to solve above-mentioned implementation process.
102) for adopting some collecting thread groups to start the every of data gathering process order data dispatching item queue placement, the step of the data gathering of business association is completed.
As shown in Fig. 1-b, this figure is collecting method entity logical relation schematic diagram based on business association described in the embodiment of the present invention, and entity specifically comprises:
1) the data item queue that data item is formed, wherein data item is collected smallest standalone unit, and as a VOD system currently provides order program service number, a current memory usage of VOD system etc. are all independent data item.
2) gatherer process (DS), based on business, describes by a four-tuple: DS=(S, T, Td, D), wherein, S represents business, and T represents data collection cycle, Td represents the data acquisition time that can accept to postpone, and T >=Td, D represents the collection of data items of associated services, and D can not be empty, that is, data item associated by a business can not be empty.
3) gatherer process queue, all DS that collection period (T) is identical merge into a gatherer process queue.
4) collecting thread group, completes the collection of a gatherer process queue.
In a word, a gatherer process associates a business, and comprises multiple data item, and a data item can be associated by multiple gatherer process, when being associated with two and above business when a data item, will be " being re-used " for the setup of attribute of this data item; A gatherer process queue comprises the identical gatherer process of multiple collection period, and a gatherer process can only belong to a queue; A collecting thread group associates one by one with a gatherer process queue.
As shown in Figure 2, this figure is the workflow diagram working as the collecting thread image data item queue that a certain collecting thread group comprises, and specifically comprises:
201) network management system starts collecting thread;
202) collecting thread reads the data item in data item queue;
203) collecting thread judges whether this data item attribute is " being re-used ", if it does, carry out 204), otherwise, carry out 205);
204) judge whether the Data Update time of this data item represents the current system time early than Tc-Td, Tc, if it does, carry out 205), otherwise, carry out 206);
205) collecting thread performs the collection of this data item;
206) judge this data item be whether data item queue last, if it does, carry out 208), otherwise, carry out 207);
207) collecting thread reads next data item in data item queue, carries out 203);
208) collecting thread waits, until next collection period arrives, carries out 202).
As shown in Figure 3, this figure is interactive television system simple relation figure, specifically comprises: a network management system server, two direct broadcast servers, two vod servers, and they may be deployed in different physical locations, between carry out interconnected with network.
Assuming that in Fig. 3 system, network management system to be completed following business datum analysis:
1) program request number of users, namely uses the number of order program service, and collection period is 60 seconds, and it is 10 seconds that the data acquisition time that can accept postpones;
2) online user number, namely uses program request and direct broadcast service total number of persons, and collection period is 60 seconds, and it is 10 seconds that the data acquisition time that can accept postpones;
3) program request flow, collection period is 10 seconds, and it is 1 second that the data acquisition time that can accept postpones;
According to of the present invention, composition graphs 3 describes, interactive television system business datum analysis entities graph of a relation as shown in Figure 4: often kind of business datum analyzes a corresponding gatherer process:
1) program request number of users gatherer process, is described as: DS1=(S1,60,10, D1), and wherein, D1 comprises vod server 1 current service logarithmic data item and vod server 2 current service logarithmic data item;
2) online user number gatherer process, is described as: DS2=(S2,60,10, D2), and wherein, D2 comprises vod server 1 current service logarithmic data item, vod server 2 current service logarithmic data item, direct broadcast server
1 current service logarithmic data item and direct broadcast server 2 current service logarithmic data item;
3) program request flow collection process, is described as: DS3=(S3,10,1, D3), and wherein, D3 comprises vod server 1 current bandwidth data item and vod server 2 current bandwidth data item;
Owing to program request number of users gatherer process is identical with the collection period of online user number gatherer process, belong to a gatherer process queue; So, in this system, comprising 2 gatherer process queues altogether, collection period is respectively 60 seconds and 10 seconds;Each gatherer process queue is completed to gather by a collecting thread group;
Thread Count according to of the present invention, included by each sets of threadsIt will be seen that Thread Count included by two collecting thread groups is all 1 in this system, that is:
The present invention is applied in network management system service data acquisition, and whole method and concrete network management system type, business type are unrelated, have ubiquity. For those skilled in the art, under the prerequisite not departing from inventive principle, the present invention is made change, also it is considered as protection scope of the present invention.
Claims (6)
1., based on a collecting method for business association, data gathering process is associated with business and provides data supporting for the network management traffic analysis for complexity by the method, and described method is:
Step 1, for generating the step of data item queue, the data item multiple to be collected belonging to same business is placed by this step continuously, and the total data item to be collected belonging to different business is placed between two kinds of business data item corresponding respectively;
Step 2, for adopting some collecting thread groups to start the every of data gathering process order data dispatching item queue placement, completes the step of the data gathering of business association;
Wherein, described data item is collected smallest standalone unit;
Described step 1 comprises following steps further:
Multiple gatherer processes identical for collection period by each data collection service corresponding gatherer process respectively, and are divided in same gatherer process queue by step 1-1;
Step 1-2, the multiple gatherer process each gatherer process queue comprised carries out gatherer process sequence according to the total relation of the data item between gatherer process;
Step 1-3, carries out data item sequence in this gatherer process by each gatherer process according to the total relation of the data item between this gatherer process and close gatherer process, obtains final data item queue;
Wherein, each gatherer process queue described corresponding collecting thread group respectively;
Described step 2 comprises following steps further:
Step 2-1, when the data gathering process started performs data gathering, first, checks whether this data item attribute is " being re-used ", if " being re-used ", enters next step; Otherwise carry out data gathering;
Step 2-2, will verify the Data Update time of this data item further, to determine whether this data item of Resurvey; If the Data Update time difference of system time and this data item is greater than the Td of gatherer process belonging to this data item, this collecting thread by this data item of Resurvey, otherwise, no longer gather; Wherein, Td represents the data acquisition time delay that can accept.
2. the collecting method based on business association according to claim 1, it is characterised in that, described gatherer process all adopts following four-tuple to describe:
DS=(S, T, Td, D);
Wherein, S represents business; T represents data collection cycle; Td represents the data acquisition time that can accept to postpone, and T >=Td; D represents the collection of data items that business packet contains, and D can not be empty, that is, data item associated by a business can not be empty, and in D, the attribute flags of the data item that different business has is " being re-used ";
Described step 1-1 judges that whether the T in the four-tuple that gatherer process is corresponding is identical, is divided in same gatherer process queue if identical by gatherer process corresponding for two business;
Described step 1-2, by empty clearly for each gatherer process queue;Gatherer process queue is inserted two gatherer process DS1、DS2, meet gatherer process DS1And DS2Compared with the data item that total data item is total with any two gatherer processes that this gatherer process queue is corresponding at most; The next gatherer process DS of insertioni, meeting current gatherer process queue top set place, length of window is Wi=q × TdiCollection of data items and gatherer process DSiTotal data item is maximum, and wherein, q represents the expectation of acquisition capacity; Repeat insertion gatherer process, until all gatherer processes are all inserted in corresponding gatherer process queue;
Described step 1-3, by each corresponding for each gatherer process " being re-used " data item, is W at front and back window widthi=q × TdiIn scope, judge to have the position of the gatherer process of this data item with this gatherer process, if front, in this gatherer process, transfer forward the position of this data item, if rear, in this gatherer process, transfer the position of this data item backward, if existed rear front simultaneously, transfer the position of this data item to centre.
3. the collecting method based on business association according to claim 2, it is characterised in that, the number of threads that described collecting thread group comprises adopts following formula to determine:
Wherein, M is the Thread Count of one group of collecting thread, and j is the business number that gatherer process queue comprises, λiFor the ageing parameter that can arrange, being worth more little, the time of collection is more short, 0 < λi≤ 1; T represents data collection cycle.
4., based on a data collecting system for business association, data gathering process is associated with business and provides data supporting for the network management traffic analysis for complexity by this system, and described system is:
Data item queue generates subsystem, the data item multiple to be collected belonging to same business is placed by this subsystem continuously, the total data item to be collected belonging to different business is placed between two business data item corresponding respectively, exports the data item queue generated; With
Data gathering subsystem, this subsystem adopts some groups of threads to start the data item queue of data gathering process order data dispatching item queue generation subsystem output, completes the data gathering of business association;
Described data item queue generates subsystem and comprises further:
Multiple gatherer processes identical for collection period for each data collection service is distinguished a corresponding gatherer process, and are divided in same gatherer process queue by gatherer process queue generation module; With
Gatherer process queue order module, multiple gatherer process for each gatherer process queue being comprised carries out gatherer process sequence according to the total relation of the data item between gatherer process, and then according to the total relation of the data item between this gatherer process and close gatherer process, each gatherer process is carried out data item sequence in this gatherer process;
Wherein, each gatherer process queue described corresponding collecting thread group respectively;
Described data gathering subsystem comprises further:
Item status inquiry and strategy generating module, for checking whether data item attribute is " being re-used ", if " being re-used ", confirm acquisition module by the input of this collection item; Otherwise carry out data gathering; With
Confirm acquisition module, for verifying the Data Update time of this data item further, to determine whether this data item of Resurvey; If the Data Update time difference of system time and this data item is greater than the Td of gatherer process belonging to this data item, this collecting thread by this data item of Resurvey, otherwise, no longer gather;Wherein, Td represents the data acquisition time delay that can accept.
5. the data collecting system based on business association according to claim 4, it is characterised in that, described gatherer process all adopts following four-tuple to describe:
DS=(S, T, Td, D);
Wherein, S represents business; T represents data collection cycle; Td represents the data acquisition time that can accept to postpone, and T >=Td; D represents the collection of data items that business packet contains, and D can not be empty, that is, data item associated by a business can not be empty, and in D, the attribute flags of the data item that different business has is " being re-used ";
Also whether T for judging in the four-tuple that gatherer process is corresponding is identical for described gatherer process queue generation module, is divided in same gatherer process queue if identical by gatherer process corresponding for two business;
Described gatherer process queue order module also for, by empty clearly for each gatherer process queue; Gatherer process queue is inserted two gatherer process DS1、DS2, meet gatherer process DS1And DS2Compared with the data item that total data item is total with any two gatherer processes that this gatherer process queue is corresponding at most; The next gatherer process DS of insertioni, meeting current gatherer process queue top set place, length of window is Wi=q × TdiCollection of data items and gatherer process DSiTotal data item is maximum, and wherein, q represents the expectation of acquisition capacity; Repeat insertion gatherer process, until all gatherer processes are all inserted in corresponding gatherer process queue;
Described gatherer process queue order module also for, by each corresponding for each gatherer process " being re-used " data item, be all W at front and back window widthi=q × TdiIn scope, judge to have the position of the gatherer process of this data item with this gatherer process, if front, in this gatherer process, transfer forward the position of this data item, if rear, in this gatherer process, transfer the position of this data item backward, if existed rear front simultaneously, transfer the position of this data item to centre.
6. the data collecting system based on business association according to claim 5, it is characterised in that, described data gathering subsystem also comprises number of threads determination module, and this module determines, according to following formula, the number of threads that each collecting thread group comprises:
Wherein, M is the Thread Count of one group of collecting thread, and j is the business number that DS queue comprises, λiFor the ageing parameter that can arrange, being worth more little, the time of collection is more short, 0 < λi≤ 1; T represents data collection cycle.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110401963.9A CN103152189B (en) | 2011-12-06 | 2011-12-06 | A kind of collecting method based on business association and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201110401963.9A CN103152189B (en) | 2011-12-06 | 2011-12-06 | A kind of collecting method based on business association and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103152189A CN103152189A (en) | 2013-06-12 |
| CN103152189B true CN103152189B (en) | 2016-06-15 |
Family
ID=48550064
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110401963.9A Expired - Fee Related CN103152189B (en) | 2011-12-06 | 2011-12-06 | A kind of collecting method based on business association and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103152189B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106549879B (en) * | 2015-09-22 | 2021-07-27 | 同方威视技术股份有限公司 | Processing method and processing device capable of automatically associating acquired data and detection system thereof |
| CN109672590A (en) * | 2019-01-10 | 2019-04-23 | 平安科技（深圳）有限公司 | Collecting method, device, equipment and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101038597A (en) * | 2007-04-29 | 2007-09-19 | 深圳市高正软件有限公司 | Data collector and system for sharing data collected |
| CN101345652A (en) * | 2008-08-15 | 2009-01-14 | 北京立通无限科技有限公司 | Data acquisition method and data acquisition equipment |
| CN101615176A (en) * | 2009-08-07 | 2009-12-30 | 北京协成致远网络科技有限公司 | A kind of data monitoring system and implementation method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100531055C (en) * | 2006-04-10 | 2009-08-19 | 华为技术有限公司 | Data synchronous system and its method |
-
2011
- 2011-12-06 CN CN201110401963.9A patent/CN103152189B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101038597A (en) * | 2007-04-29 | 2007-09-19 | 深圳市高正软件有限公司 | Data collector and system for sharing data collected |
| CN101345652A (en) * | 2008-08-15 | 2009-01-14 | 北京立通无限科技有限公司 | Data acquisition method and data acquisition equipment |
| CN101615176A (en) * | 2009-08-07 | 2009-12-30 | 北京协成致远网络科技有限公司 | A kind of data monitoring system and implementation method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103152189A (en) | 2013-06-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102955977B (en) | Energy efficiency service method and energy efficiency service platform adopting same on basis of cloud technology | |
| CN104038540B (en) | Method and system for automatically selecting application proxy server | |
| CN109815011A (en) | A kind of method and apparatus of data processing | |
| CN102929961A (en) | Data processing method and device thereof based on building quick data staging channel | |
| CN105306557A (en) | Bridge health monitoring system based on cloud platform | |
| CN108428114A (en) | Information processing method and device, storage medium, the electronic device of project | |
| CN107844922A (en) | residential electricity consumption scheduling system based on cloud computing platform | |
| CN106502875A (en) | A kind of daily record generation method and system based on cloud computing | |
| CN104778528A (en) | Method and system for obtaining smart city evaluation indexes by utilizing big data | |
| CN103152189B (en) | A kind of collecting method based on business association and system | |
| CN102404760B (en) | Method and device for real-time measurement of system performance | |
| CN107391606A (en) | Log processing method and device based on Storm | |
| CN101252480B (en) | Realizing method for establishing multimedia webpage browsing traffic model | |
| CN106202399A (en) | Method for implementing data management system of big data | |
| CN108520329A (en) | The accurate auto-allocation method of second-hand house client and device based on broker's portrait | |
| CN106446084A (en) | Internet person search information integration analysis method | |
| CN112216061A (en) | Rainwater condition monitoring and early warning method and system | |
| CN106130929A (en) | The service message automatic processing method of the Internet based on graph-theoretical algorithm insurance field and system | |
| CN105450453A (en) | Network topology construction method based on SNMP | |
| CN109375146A (en) | A kind of filling mining method, system and the terminal device of electricity consumption data | |
| CN104714875A (en) | Distributed automatic collecting method | |
| CN105224327A (en) | A kind of embedded equipment variable exchanges transport address off-line distribution method | |
| CN106790474A (en) | A kind of energy data real-time acquisition system and acquisition method | |
| CN112001772A (en) | Distributed data storage method with screening structure for block chain electric commerce | |
| CN106909459A (en) | A kind of method and device for adjusting connection pool |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20160615 |