CN105281967A - Data collection method and system - Google Patents
Data collection method and system Download PDFInfo
- Publication number
- CN105281967A CN105281967A CN201410280346.1A CN201410280346A CN105281967A CN 105281967 A CN105281967 A CN 105281967A CN 201410280346 A CN201410280346 A CN 201410280346A CN 105281967 A CN105281967 A CN 105281967A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- acquisition probe
- probe
- data
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Medical Treatment And Welfare Office Work (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention discloses a data collection method and system, and relates to the field of computer network technology. Via dynamical monitoring, survival data processing centers and collection probes are dynamically searched to implement corresponding collection when a data processing center is disconnected from a collection probe, and the problem that the whole data collection task cannot be carried out correctly due to stop of the collection probe or data processing center for certain reason is solved. Via load balancing, multiple collection probes can be used to implement collection when multiple devices are collected in a batch manner, and compared with a single collection probe in the prior art, the efficiency can be improved. Compared with a single-collection-probe mechanism in the prior art, collection is prevented from stop caused by that the collection probe is unavailable due to the network and the like.
Description
Technical field
The present invention relates to technical field of the computer network, particularly a kind of collecting method and system.
Background technology
Along with the development of Internet technology, and the enriching constantly of network application demand, the application of data acquisition gets more and more, and also more and more important, the application such as the network element polling of such as data processing centre, all relate to corresponding data acquisition operations on each network element.Therefore, the efficiency of data acquisition and accuracy most important.
In prior art, the primary solutions of data acquisition is, configures an acquisition probe at each network element end, when needs perform data acquisition session, data processing centre decomposes acquisition tasks, gathers required data by acquisition probe from network element to be collected.
But, in existing data acquisition technology, because acquisition probe and data processing centre are unaware for backstage, if certain data processing centre or acquisition probe are due to reasons such as networks, cannot network be connected to, namely be in non-viable state, the state that cannot normally connect, the problem such as then can cause data acquisition failure or make a mistake, affects the accuracy of data acquisition.In addition, if certain period is too much for the data acquisition session of a certain network element, then the acquisition probe pressure of its correspondence can be caused excessive, not only affect the efficiency of data acquisition, also easily cause the phenomenons such as collapse because acquisition probe pressure is excessive, finally cause data acquisition failure or make mistakes.Given this, in the urgent need to a kind of more accurately and googol can be tolerated according to the data acquisition solution gathering pressure, to be suitable for day by day urgent data acquisition demand.
Summary of the invention
In view of the above problems, the embodiment of the present invention provides a kind of collecting method and system, to solve inaccurate in available data acquisition scheme and cannot bear the problem of huge data acquisition pressure.
The embodiment of the present invention have employed following technical scheme:
One embodiment of the invention provides a kind of collecting method, and described method comprises:
For each network element arranges acquisition probe pond, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe;
When data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.
It is multiple for obtaining according to snoop results the acquisition probe being in existing state in network element to be collected correspondence collection pond, and described method also comprises:
Assess multiple load state being in the acquisition probe of existing state, according to load balancing mode determination preferred acquisition probe;
Described utilization is in the data processing centre of existing state and acquisition probe and carries out data acquisition session and be specially:
Utilize and be in the data processing centre of existing state and described preferred acquisition probe carries out data acquisition session.
The multiple load state being in the acquisition probe of existing state of described assessment, comprises according to load balancing mode determination preferred acquisition probe:
For each acquisition probe, calculate the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, obtain the score value of the load state of each acquisition probe;
Using acquisition probe the highest for score value as preferred acquisition probe.
Described method also comprises:
After a data acquisition operations terminates, the collection retaining this data acquisition connects; Described collection connects for the network connection relation between the data processing centre for performing the foundation of this data acquisition operations, acquisition probe and this data acquisition network element;
If new data acquisition operations belongs to same acquisition tasks with data acquisition operations before, whether the collection of the data acquisition before then judging in the collection retained connects connects within valid expiration date, if within valid expiration date, the collection of the data acquisition before then directly adopting connects carries out this new data acquisition operations, if not in valid expiration date, then perform the described data processing centre being in existing state according to snoop results acquisition, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize and be in the data processing centre of existing state and acquisition probe carries out data acquisition session.
Describedly acquisition probe pond be set comprise for each network element:
According to the collection conditions of demand of network element, determine the acquisition probe quantity of the required configuration of each network element;
For each network element configuration comprises the acquisition probe pond of the acquisition probe quantity determined.
Described method also comprises: the collection demand according to network element changes, the corresponding acquisition probe quantity changing network element;
When the acquisition probe in acquisition probe pond is newly-increased or when restarting, acquisition probe that is newly-increased or that restart starts one first monitoring service, connection request is sent to data processing centre, after described data processing centre receives connection request, start the second monitoring service with this acquisition probe, periodically monitor the connection status with this acquisition probe;
When the acquisition probe in acquisition probe pond reduces, data processing centre monitors not then the acquisition probe of this minimizing, then close the second monitoring service of the acquisition probe to this minimizing.
In addition, the embodiment of the present invention additionally provides a kind of data acquisition system, and described system comprises:
Arrange and monitor module, for arranging acquisition probe pond for each network element, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe;
Data acquisition module, for when data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.
Described data acquisition module also comprises:
Load evaluation unit, when being multiple for obtaining the acquisition probe being in existing state in network element to be collected correspondence collection pond according to snoop results, assess multiple load state being in the acquisition probe of existing state, according to load balancing mode determination preferred acquisition probe;
Described load evaluation unit comprises:
Scoring subelement, for for each acquisition probe, calculates the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, obtains the score value of the load state of each acquisition probe;
Determine subelement, for using acquisition probe the highest for score value as preferred acquisition probe.
Described data acquisition module also comprises:
Connect storage unit, for after a data acquisition operations terminates, the collection retaining this data acquisition connects; Described collection connects for the network connection relation between the data processing centre for performing the foundation of this data acquisition operations, acquisition probe and this data acquisition network element;
Judging unit, if for new data acquisition operations with described be connected that storage unit preserves before data acquisition operations belong to same acquisition tasks, whether the collection of the data acquisition before then judging in the collection retained connects connects within valid expiration date, if within valid expiration date, the collection of the data acquisition before then directly adopting connects carries out this new data acquisition operations, if not in valid expiration date, then perform the described data processing centre being in existing state according to snoop results acquisition, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize and be in the data processing centre of existing state and acquisition probe carries out data acquisition session.
Described setting and monitoring module comprise:
Acquisition probe pond determining unit, for the collection conditions of demand according to network element, determines the acquisition probe quantity of the required configuration of each network element;
Acquisition probe pond setting unit, for comprising the acquisition probe pond of the acquisition probe quantity determined for each network element configuration;
Monitoring unit, for each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe;
Described setting and monitoring module also comprise:
Changing unit, changes for the collection demand according to network element, the corresponding acquisition probe quantity changing network element;
Adding device, for when the acquisition probe in acquisition probe pond is newly-increased or when restarting, acquisition probe that is newly-increased or that restart starts one first monitoring service, connection request is sent to data processing centre, after described data processing centre receives connection request, start the second monitoring service with this acquisition probe, periodically monitor the connection status with this acquisition probe;
Delete cells, for when the acquisition probe in acquisition probe pond reduces, data processing centre monitors not then the acquisition probe of this minimizing, then close the second monitoring service of the acquisition probe to this minimizing.
A kind of collecting method that the embodiment of the present invention provides and system, for each network element arranges acquisition probe pond, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe; When data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.Visible, the embodiment of the present invention is by dynamically monitoring, can ensure that data processing centre and acquisition probe are when disconnecting, dynamically look for the data processing centre of survival and acquisition probe to carry out corresponding acquisition process, will, because of acquisition probe or data processing centre because certain reason stops, whole data acquisition session be caused correctly to perform.
Further, in the embodiment of the present invention, by load balancing, can, when batch capture multiple devices, use multiple acquisition probe dynamically to carry out Collecting operation, more efficiency can be improved compared with the single acquisition probe of prior art.Further, compared to prior art list acquisition probe mechanism, can prevent from, because reasons such as networks, causing the acquisition probe occurring Equipments Setting to be in disabled state, causing Collecting operation to stop.
Accompanying drawing explanation
A kind of collecting method flow chart that Fig. 1 embodiment of the present invention provides;
Fig. 2 is data acquisition network structural representation in the embodiment of the present invention;
A kind of data acquisition system structured flowchart that Fig. 3 provides for the embodiment of the present invention.
Embodiment
For making the object, technical solutions and advantages of the present invention clearly, below in conjunction with accompanying drawing, embodiment of the present invention is described further in detail.
See Fig. 1, the embodiment of the present invention provides a kind of method of data acquisition, mainly comprises following concrete steps:
S101: for each network element arranges acquisition probe pond, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe.
It should be noted that, the embodiment of the present invention can be, but not limited to be applied to RIM system and carries out data acquisition.
RMI (RemoteMethodInvocation, RMI) realizes in JDK1.1 with Java, and it greatly strengthen the ability of Java development DDB application.Communicate each other between the program level object that RMI supports to be stored in different address space, realize the seamless far call between remote object.
Shown in Figure 2, for arranging acquisition probe pond schematic diagram according to step of the present invention.Here, network element for monitored database, be managed main frame, other network equipment etc. and illustrate.
In order to keep the connection of acquisition probe and data processing centre, acquisition probe end and data processing centre's end start monitoring respectively, at set intervals, acquisition probe or data processing centre can go dynamically to find current can interconnective data processing centre or acquisition probe with it, preserve after connecting, acquisition probe pond can be used carry out the different acquisition probe of dynamic use to gather for different resources, the correct running of data processing centre and acquisition probe can be ensured, prevent because the reasons such as network cause Joint failure, the data processing centre of inefficacy or acquisition probe is still adopted to carry out Mission Operations, make the mistake.
A kind of concrete implementation can be:
1, acquisition probe end distributes a fritter internal memory to preserve all data processing centres of all associateds.
2, data processing centre's end distributes a fritter internal memory to preserve all acquisition probe of all associateds.
3, acquisition probe is when starting, and can go to initiate and the connection request of data processing centre, asks by being then kept in the internal memory that oneself distributes.Further, data processing centre, when starting, can go to initiate and the connection request of acquisition probe, asks by being then kept in the internal memory that oneself distributes.
4, at set intervals, whether the monitor of data processing centre will go to search the acquisition probe of preserving in internal memory and can use, same, whether the monitor of acquisition probe will go to search the data processing centre preserved in internal memory and can use, guarantee that data acquisition service is all correctly available, avoid mistake.In practical application, heartbeat mode property performance period can be adopted to monitor.
That is, the new mechanism proposed in the embodiment of the present invention can ensure that the data processing centre that existed and acquisition probe are available, the other side can be deleted by monitoring mechanism by disabled service each other from the internal memory of oneself, only use existing, connection existence, ensure the correct running of data processing centre and acquisition probe.
As preferably, describedly to comprise for each network element arranges acquisition probe pond:
According to the collection conditions of demand of network element, determine the acquisition probe quantity of the required configuration of each network element;
For each network element configuration comprises the acquisition probe pond of the acquisition probe quantity determined.
That is, arranging in the step in acquisition probe pond for each network element, preferably arrange according to the collection conditions of demand of each network element, for in system, the data acquisition demand of some network element is little, namely, data acquisition operations in this network element is relatively less, the data acquisition demand of some network element is but very large, namely the data acquisition operations in this network element is relatively frequent, then for the network element that data acquisition demand is few, for acquisition probe quantity included in the acquisition probe pond that it is arranged is relatively less, for the network element that data acquisition demand is many, for acquisition probe quantity included in the acquisition probe pond that it is arranged is relatively many.
Further, described in the embodiment of the present invention, method also comprises: the collection demand according to network element changes, the corresponding acquisition probe quantity changing network element.
Concrete, when the acquisition probe in acquisition probe pond reduces, data processing centre monitors not then the acquisition probe of this minimizing, then close the second monitoring service of the acquisition probe to this minimizing.Namely follow-up without the need to monitoring it again.In practical application, can be in oneself internal memory, the acquisition probe of this minimizing is deleted.
When the acquisition probe in acquisition probe pond is newly-increased or when restarting, acquisition probe that is newly-increased or that restart starts one first monitoring service, connection request is sent to data processing centre, after described data processing centre receives connection request, start the second monitoring service with this acquisition probe, periodically monitor the connection status with this acquisition probe.Namely follow-uply will periodically it to be monitored.In practical application, acquisition probe that is this is newly-increased or that restart is added in oneself internal memory.
S102: when data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.
Preferably, it is multiple for obtaining according to snoop results the acquisition probe being in existing state in network element to be collected correspondence collection pond, and described method also comprises:
Assess multiple load state being in the acquisition probe of existing state, according to load balancing mode determination preferred acquisition probe.
Accordingly, described utilization is in the data processing centre of existing state and acquisition probe and carries out data acquisition session and be specially:
Utilize and be in the data processing centre of existing state and described preferred acquisition probe carries out data acquisition session.
As a kind of preferred version, the multiple load state being in the acquisition probe of existing state of described assessment, comprises according to load balancing mode determination preferred acquisition probe:
For each acquisition probe, calculate the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, obtain the score value of the load state of each acquisition probe;
Using acquisition probe the highest for score value as preferred acquisition probe.
That is, use the load-balancing mechanism in acquisition probe pond, can ensure in the process of implementation, can distribute acquisition probe dynamically and carry out image data, the acquisition probe that selection pressure is little carries out data acquisition, can ensure the correct collection of data.
A concrete load state appraisal procedure, for each acquisition probe, calculate the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, the score value obtaining the load state of each acquisition probe can be:
The load state score value of acquisition probe=
Current C PU utilization rate * utilance weight coefficient+current memory utilization rate * memory usage weight coefficient+storage allocation space * memory headroom weight coefficient
That is, the embodiment of the present invention proposes a kind of concrete load state assessment algorithm, i.e. in good time system scoring algorithm: the performance index that this algorithm is mainly used to describe the acquisition probe that current server is disposed are shown, by CPU and the current utilization rate (obtaining the current CP that marks according to master frequency and current C PU utilization rate) of timing acquisition server, internal memory and current utilization rate (obtain current free memory and obtain the current MP that marks), start the internal memory that when serving, JVM distributes and (obtain the stack memory that current JVM distributes, obtain JVM configuration scoring JP) etc. information, calculate the index comprehensive assessment numerical value of each acquisition probe.
In practical application, in order to improve the accuracy of assessment further, load state assessment algorithm can be judge whether current C PU utilization rate CP, current memory utilization rate MP and storage allocation space JP tri-values differ greatly, such as utilize maximum in above-mentioned three values divided by minimum value, judge whether ratio exceedes preset value (supposing can judge whether to be greater than 120%), if be greater than, then illustrate that difference is larger, if be less than, then illustrate and be more or less the same.
If the maximum in CP, MP, JP tri-value is divided by minimum value, result exceedes preset value, then illustrate that CP, MP, JP tri-value differences are larger, in order to reduce error further, minimum value in CP, MP, JP tri-values is adopted to assess, preferably, utilance weight coefficient is set to 100%, and memory usage weight coefficient and memory headroom weight coefficient are set to 0%.
If the maximum in CP, MP, JP tri-value is divided by minimum value, result does not exceed preset value, then illustrate that CP, MP, JP tri-values are more or less the same, corresponding weight coefficient can be set according to the influence degree of three values to load state, preferably, utilance weight coefficient is set to 40%, and memory usage weight coefficient is set to 40%, and memory headroom weight coefficient is set to 20%.
Illustrate: for each probe, in CP, MP, JP tri-value, maximum divided by minimum, if be greater than 120%, then use the scoring as this probe minimum in three; If maximum is less than 120% divided by minimum, then use CP*40%+MP*40%+JP*20% as the scoring of this probe.
Load balancing adopts algorithm, examination can be carried out for probe that is least-recently-used in acquisition probe, that performed minimum task to use, adopt dynamic assignment, ensure to a certain extent, ensure that acquisition probe selected by resource is the minimum acquisition probe of current pressure.
Further, described in the embodiment of the present invention, method also comprises:
After a data acquisition operations terminates, the collection retaining this data acquisition connects; Described collection connects for the network connection relation between the data processing centre for performing the foundation of this data acquisition operations, acquisition probe and this data acquisition network element.
If new data acquisition operations belongs to same acquisition tasks with data acquisition operations before, whether the collection of the data acquisition before then judging in the collection retained connects connects within valid expiration date, if within valid expiration date, the collection of the data acquisition before then directly adopting connects carries out this new data acquisition operations, if not in valid expiration date, then perform the described data processing centre being in existing state according to snoop results acquisition, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize and be in the data processing centre of existing state and acquisition probe carries out data acquisition session.
That is, a data acquisition session performs realization by multiple data acquisition operations, after a data acquisition operations terminates, preserve the connection that this data acquisition operations uses, if follow-up data acquisition operations belongs to same data acquisition session with data acquisition operations before, the connection that before then searching in the connection of preserving, data acquisition operations uses, judge whether within valid expiration date, if, then without the need to again assessing the load state of each acquisition probe, and re-establish connection, improve data acquisition efficiency further.If not in valid expiration date, then the load state of each acquisition probe that needs to reappraise, and re-establish data acquisition and connect.
A kind of collecting method that the embodiment of the present invention provides, for each network element arranges acquisition probe pond, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe; When data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.Visible, the embodiment of the present invention is by dynamically monitoring, can ensure that data processing centre and acquisition probe are when disconnecting, dynamically look for the data processing centre of survival and acquisition probe to carry out corresponding acquisition process, will because of acquisition probe or data processing centre because certain reason stops, and whole data acquisition session be caused correctly to perform.
Further, in the embodiment of the present invention, by load balancing, can, when batch capture multiple devices, use multiple acquisition probe dynamically to carry out Collecting operation, more efficiency can be improved compared with the single acquisition probe of prior art.Further, compared to prior art list acquisition probe mechanism, can prevent from, because reasons such as networks, causing the acquisition probe occurring Equipments Setting to be in disabled state, causing Collecting operation to stop.
See Fig. 3, and in conjunction with shown in above-mentioned Fig. 2, the embodiment of the present invention provides a kind of data acquisition system, and described system comprises:
Arrange and monitor module 301, for arranging acquisition probe pond for each network element, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe.
Data acquisition module 302, for when data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.
Preferably, described data acquisition module also comprises:
Load evaluation unit, when being multiple for obtaining the acquisition probe being in existing state in network element to be collected correspondence collection pond according to snoop results, assess multiple load state being in the acquisition probe of existing state, according to load balancing mode determination preferred acquisition probe.
Concrete, described load evaluation unit comprises:
Scoring subelement, for for each acquisition probe, calculates the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, obtains the score value of the load state of each acquisition probe;
Determine subelement, for using acquisition probe the highest for score value as preferred acquisition probe.
Further, described data acquisition module also comprises:
Connect storage unit, for after a data acquisition operations terminates, the collection retaining this data acquisition connects; Described collection connects for the network connection relation between the data processing centre for performing the foundation of this data acquisition operations, acquisition probe and this data acquisition network element;
Judging unit, if for new data acquisition operations with described be connected that storage unit preserves before data acquisition operations belong to same acquisition tasks, whether the collection of the data acquisition before then judging in the collection retained connects connects within valid expiration date, if within valid expiration date, the collection of the data acquisition before then directly adopting connects carries out this new data acquisition operations, if not in valid expiration date, then perform the described data processing centre being in existing state according to snoop results acquisition, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize and be in the data processing centre of existing state and acquisition probe carries out data acquisition session.
As concrete, described setting and monitoring module comprise:
Acquisition probe pond determining unit, for the collection conditions of demand according to network element, determines the acquisition probe quantity of the required configuration of each network element;
Acquisition probe pond setting unit, for comprising the acquisition probe pond of the acquisition probe quantity determined for each network element configuration;
Monitoring unit, for each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe.
Preferably, described setting and monitoring module also comprise:
Changing unit, changes for the collection demand according to network element, the corresponding acquisition probe quantity changing network element;
Adding device, for when the acquisition probe in acquisition probe pond is newly-increased or when restarting, acquisition probe that is newly-increased or that restart starts one first monitoring service, connection request is sent to data processing centre, after described data processing centre receives connection request, start the second monitoring service with this acquisition probe, periodically monitor the connection status with this acquisition probe;
Delete cells, for when the acquisition probe in acquisition probe pond reduces, data processing centre monitors not then the acquisition probe of this minimizing, then close the second monitoring service of the acquisition probe to this minimizing.
It should be noted that, the modules in present system embodiment or the operation principle of unit and processing procedure see the associated description in embodiment of the method shown in above-mentioned Fig. 1-Fig. 2, can repeat no more herein.
A kind of data acquisition system that the embodiment of the present invention provides, for each network element arranges acquisition probe pond, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe; When data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.Visible, the embodiment of the present invention is by dynamically monitoring, can ensure that data processing centre and acquisition probe are when disconnecting, dynamically look for the data processing centre of survival and acquisition probe to carry out corresponding acquisition process, will because of acquisition probe or data processing centre because certain reason stops, and whole data acquisition session be caused correctly to perform.
Further, in the embodiment of the present invention, by load balancing, can, when batch capture multiple devices, use multiple acquisition probe dynamically to carry out Collecting operation, more efficiency can be improved compared with the single acquisition probe of prior art.Further, compared to prior art list acquisition probe mechanism, can prevent from, because reasons such as networks, causing the acquisition probe occurring Equipments Setting to be in disabled state, causing Collecting operation to stop.
For the ease of the technical scheme of the clear description embodiment of the present invention, in inventive embodiment, have employed the printed words such as " first ", " second " to distinguish the substantially identical identical entry of function and efficacy or similar item, it will be appreciated by those skilled in the art that the printed words such as " first ", " second " do not limit quantity and execution order.
One of ordinary skill in the art will appreciate that, the all or part of step realized in above-described embodiment method is that the hardware that can carry out instruction relevant by program has come, described program can be stored in a computer read/write memory medium, this program is when performing, comprise the steps: (step of method), described storage medium, as: ROM/RAM, magnetic disc, CD etc.
The foregoing is only preferred embodiment of the present invention, be not intended to limit protection scope of the present invention.All any amendments done within the spirit and principles in the present invention, equivalent replacement, improvement etc., be all included in protection scope of the present invention.
Claims (10)
1. a collecting method, is characterized in that, described method comprises:
For each network element arranges acquisition probe pond, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe;
When data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.
2. collecting method according to claim 1, is characterized in that, it is multiple for obtaining according to snoop results the acquisition probe being in existing state in network element to be collected correspondence collection pond, and described method also comprises:
Assess multiple load state being in the acquisition probe of existing state, according to load balancing mode determination preferred acquisition probe;
Described utilization is in the data processing centre of existing state and acquisition probe and carries out data acquisition session and be specially:
Utilize and be in the data processing centre of existing state and described preferred acquisition probe carries out data acquisition session.
3. collecting method according to claim 2, is characterized in that, the multiple load state being in the acquisition probe of existing state of described assessment, comprises according to load balancing mode determination preferred acquisition probe:
For each acquisition probe, calculate the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, obtain the score value of the load state of each acquisition probe;
Using acquisition probe the highest for score value as preferred acquisition probe.
4. collecting method according to claim 1, is characterized in that, described method also comprises:
After a data acquisition operations terminates, the collection retaining this data acquisition connects; Described collection connects for the network connection relation between the data processing centre for performing the foundation of this data acquisition operations, acquisition probe and this data acquisition network element;
If new data acquisition operations belongs to same acquisition tasks with data acquisition operations before, whether the collection of the data acquisition before then judging in the collection retained connects connects within valid expiration date, if within valid expiration date, the collection of the data acquisition before then directly adopting connects carries out this new data acquisition operations, if not in valid expiration date, then perform the described data processing centre being in existing state according to snoop results acquisition, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize and be in the data processing centre of existing state and acquisition probe carries out data acquisition session.
5. collecting method according to claim 1, is characterized in that, describedly arranges acquisition probe pond and comprises for each network element:
According to the collection conditions of demand of network element, determine the acquisition probe quantity of the required configuration of each network element;
For each network element configuration comprises the acquisition probe pond of the acquisition probe quantity determined.
6. collecting method according to claim 5, is characterized in that, described method also comprises: the collection demand according to network element changes, the corresponding acquisition probe quantity changing network element;
When the acquisition probe in acquisition probe pond is newly-increased or when restarting, acquisition probe that is newly-increased or that restart starts one first monitoring service, connection request is sent to data processing centre, after described data processing centre receives connection request, start the second monitoring service with this acquisition probe, periodically monitor the connection status with this acquisition probe;
When the acquisition probe in acquisition probe pond reduces, data processing centre monitors not then the acquisition probe of this minimizing, then close the second monitoring service of the acquisition probe to this minimizing.
7. a data acquisition system, is characterized in that, described system comprises:
Arrange and monitor module, for arranging acquisition probe pond for each network element, each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe;
Data acquisition module, for when data acquisition session need be carried out, the data processing centre being in existing state is obtained according to snoop results, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize the data processing centre that is in existing state and acquisition probe to carry out data acquisition session.
8. data acquisition system according to claim 7, is characterized in that, described data acquisition module also comprises:
Load evaluation unit, when being multiple for obtaining the acquisition probe being in existing state in network element to be collected correspondence collection pond according to snoop results, assess multiple load state being in the acquisition probe of existing state, according to load balancing mode determination preferred acquisition probe;
Described load evaluation unit comprises:
Scoring subelement, for for each acquisition probe, calculates the weighting sum in current C PU utilization rate, current memory utilization rate and storage allocation space, obtains the score value of the load state of each acquisition probe;
Determine subelement, for using acquisition probe the highest for score value as preferred acquisition probe.
9. data acquisition system according to claim 7, is characterized in that, described data acquisition module also comprises:
Connect storage unit, for after a data acquisition operations terminates, the collection retaining this data acquisition connects; Described collection connects for the network connection relation between the data processing centre for performing the foundation of this data acquisition operations, acquisition probe and this data acquisition network element;
Judging unit, if for new data acquisition operations with described be connected that storage unit preserves before data acquisition operations belong to same acquisition tasks, whether the collection of the data acquisition before then judging in the collection retained connects connects within valid expiration date, if within valid expiration date, the collection of the data acquisition before then directly adopting connects carries out this new data acquisition operations, if not in valid expiration date, then perform the described data processing centre being in existing state according to snoop results acquisition, and in the corresponding acquisition probe pond of network element to be collected, be in the acquisition probe of existing state, utilize and be in the data processing centre of existing state and acquisition probe carries out data acquisition session.
10. data acquisition system according to claim 7, is characterized in that, described setting and monitoring module comprise:
Acquisition probe pond determining unit, for the collection conditions of demand according to network element, determines the acquisition probe quantity of the required configuration of each network element;
Acquisition probe pond setting unit, for comprising the acquisition probe pond of the acquisition probe quantity determined for each network element configuration;
Monitoring unit, for each acquisition probe each self-starting first monitoring service in acquisition probe pond, periodically monitor the connection status with data processing centre, and the heart starts the second monitoring service in data handling, periodically monitor the connection status with each acquisition probe;
Described setting and monitoring module also comprise:
Changing unit, changes for the collection demand according to network element, the corresponding acquisition probe quantity changing network element;
Adding device, for when the acquisition probe in acquisition probe pond is newly-increased or when restarting, acquisition probe that is newly-increased or that restart starts one first monitoring service, connection request is sent to data processing centre, after described data processing centre receives connection request, start the second monitoring service with this acquisition probe, periodically monitor the connection status with this acquisition probe;
Delete cells, for when the acquisition probe in acquisition probe pond reduces, data processing centre monitors not then the acquisition probe of this minimizing, then close the second monitoring service of the acquisition probe to this minimizing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410280346.1A CN105281967B (en) | 2014-06-20 | 2014-06-20 | A kind of collecting method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410280346.1A CN105281967B (en) | 2014-06-20 | 2014-06-20 | A kind of collecting method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105281967A true CN105281967A (en) | 2016-01-27 |
| CN105281967B CN105281967B (en) | 2019-06-28 |
Family
ID=55150337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410280346.1A Active CN105281967B (en) | 2014-06-20 | 2014-06-20 | A kind of collecting method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105281967B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311878A (en) * | 2020-10-30 | 2021-02-02 | 杭州迪普科技股份有限公司 | Centralized control probe disconnection reconnection method, device and system |
| CN114007232A (en) * | 2020-12-14 | 2022-02-01 | 北京八分量信息科技有限公司 | Keep-alive mechanism realized by optimizing node communication |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6278694B1 (en) * | 1999-04-16 | 2001-08-21 | Concord Communications Inc. | Collecting and reporting monitoring data from remote network probes |
| GB2371442A (en) * | 2000-11-01 | 2002-07-24 | Agilent Technologies Inc | Monitoring communication network configuration by probing and characterising network data streams |
| CN1960274A (en) * | 2005-11-01 | 2007-05-09 | 中兴通讯股份有限公司 | Method for dynamic real time capturing logic commands input from UNIX terminal user |
| CN101201782A (en) * | 2006-12-01 | 2008-06-18 | 株式会社日立制作所 | Debug information collection method and debug information collection system |
| CN102143022A (en) * | 2011-03-16 | 2011-08-03 | 北京邮电大学 | Cloud measurement device and method for IP network |
| CN102801587A (en) * | 2012-08-29 | 2012-11-28 | 北京邮电大学 | Large-scale network-oriented virtualized monitoring system and dynamic monitoring method thereof |
-
2014
- 2014-06-20 CN CN201410280346.1A patent/CN105281967B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6278694B1 (en) * | 1999-04-16 | 2001-08-21 | Concord Communications Inc. | Collecting and reporting monitoring data from remote network probes |
| GB2371442A (en) * | 2000-11-01 | 2002-07-24 | Agilent Technologies Inc | Monitoring communication network configuration by probing and characterising network data streams |
| CN1960274A (en) * | 2005-11-01 | 2007-05-09 | 中兴通讯股份有限公司 | Method for dynamic real time capturing logic commands input from UNIX terminal user |
| CN101201782A (en) * | 2006-12-01 | 2008-06-18 | 株式会社日立制作所 | Debug information collection method and debug information collection system |
| CN102143022A (en) * | 2011-03-16 | 2011-08-03 | 北京邮电大学 | Cloud measurement device and method for IP network |
| CN102801587A (en) * | 2012-08-29 | 2012-11-28 | 北京邮电大学 | Large-scale network-oriented virtualized monitoring system and dynamic monitoring method thereof |
Non-Patent Citations (1)
| Title |
|---|
| 姜志: "《IPTV性能测量探针的设计与实现》", 《中国优秀硕士学位论文全文数据库（信息科技辑）》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311878A (en) * | 2020-10-30 | 2021-02-02 | 杭州迪普科技股份有限公司 | Centralized control probe disconnection reconnection method, device and system |
| CN114007232A (en) * | 2020-12-14 | 2022-02-01 | 北京八分量信息科技有限公司 | Keep-alive mechanism realized by optimizing node communication |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105281967B (en) | 2019-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11429506B2 (en) | Systems and methods for collecting, tracking, and storing system performance and event data for computing devices | |
| CN105357038B (en) | Monitor the method and system of cluster virtual machine | |
| US8924666B2 (en) | Managing storage devices in a cloud storage environment | |
| CN112231075B (en) | Cloud service-based server cluster load balancing control method and system | |
| CN110865867A (en) | Method, device and system for discovering application topological relation | |
| US20140143625A1 (en) | Computer-readable recording medium, failure prediction device and applicability determination method | |
| CN106506269B (en) | It executes the method for test assignment, system, calculate equipment and test macro | |
| CN110659109A (en) | Openstack cluster virtual machine monitoring system and method | |
| CN111625419A (en) | Log acquisition method, system, equipment and computer readable storage medium | |
| CN111526038B (en) | Service request distribution method and device, computer equipment and readable storage medium | |
| CN105281967A (en) | Data collection method and system | |
| CN107483280B (en) | Method and device for monitoring service node device | |
| CN117632897A (en) | Dynamic capacity expansion and contraction method and device | |
| CN108200151A (en) | ISCSI Target load-balancing methods and device in a kind of distributed memory system | |
| CN112685486B (en) | Data management method and device for database cluster, electronic equipment and storage medium | |
| CN116107741A (en) | Load balancing scheduling method, system, electronic equipment and readable storage medium | |
| CN115562933A (en) | Processing method and device of operation monitoring data, storage medium and electronic equipment | |
| TWI579691B (en) | Method and System of IDC Computer Room Entity and Virtual Host Integration Management | |
| CN110213106B (en) | Equipment information management method, device and system and electronic equipment | |
| CN110247802B (en) | Resource configuration method and device for cloud service single-machine environment | |
| CN112527469A (en) | Fault-tolerant combination method of cloud computing server | |
| CN112968933B (en) | Data transmission method, device, server and storage medium | |
| CN112667460A (en) | Method for monitoring avionics system application task stack space | |
| CN111901174A (en) | Service state notification method, related device and storage medium | |
| US20220200879A1 (en) | Managing stateful workloads executing on temporarily available resources of a cloud computing system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 818, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after: BEIJING ULTRAPOWER SOFTWARE Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building 6 storey block A Room 601 Patentee before: BEIJING ULTRAPOWER SOFTWARE Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |