CN105281967B - A kind of collecting method and system - Google Patents
A kind of collecting method and system Download PDFInfo
- Publication number
- CN105281967B CN105281967B CN201410280346.1A CN201410280346A CN105281967B CN 105281967 B CN105281967 B CN 105281967B CN 201410280346 A CN201410280346 A CN 201410280346A CN 105281967 B CN105281967 B CN 105281967B
- Authority
- CN
- China
- Prior art keywords
- acquisition
- probe
- acquisition probe
- data
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Measuring And Recording Apparatus For Diagnosis (AREA)
- Medical Treatment And Welfare Office Work (AREA)
Abstract
The invention discloses a kind of collecting method and systems, are related to technical field of the computer network.The embodiment of the present invention is monitored by dynamic, data processing centre and acquisition probe can be guaranteed when disconnecting, dynamic looks for data processing centre and the acquisition probe of survival to carry out corresponding acquisition process, entire data acquisition session will not be caused not execute correctly because of acquisition probe or data processing centre because certain reason stops.Further, in the embodiment of the present invention, by load balancing, it can be acquired operation using multiple acquisition probe dynamics, more efficiency can be improved compared with the single acquisition probe of the prior art in batch capture multiple devices.Also, it can prevent for single acquisition probe mechanism because the reasons such as network, cause the acquisition probe for device configuration occur to be in unusable state, Collecting operation is caused to stop compared with the prior art.
Description
Technical field
The present invention relates to technical field of the computer network, in particular to a kind of collecting method and system.
Background technique
With the continuous development of Internet technology and enriching constantly for network application demand, the application of data acquisition is got over
Come it is more, it is also more and more important, such as the application such as network element polling of data processing centre, be directed to respective counts on each network element
It is operated according to acquisition.Therefore, the efficiency and accuracy of data acquisition is most important.
In the prior art, the primary solutions of data acquisition are to configure an acquisition probe at each network element end, when needs are held
When row data acquisition session, data processing centre decomposes acquisition tasks, is adopted from network element to be collected by acquisition probe
Data required for collecting.
But in existing data acquisition technology, since acquisition probe and data processing centre are for being unaware from the background
, if certain data processing centre or acquisition probe cannot connect to network due to network etc., that is, it is in non-viable state,
The state that can not normally connect influences the accuracy of data acquisition the problems such as then will lead to data acquisition failure or mistake occurs.
In addition, will lead to its corresponding acquisition probe pressure mistake if certain time is excessive for the data acquisition session of a certain network element
Greatly, the efficiency of data acquisition is not only influenced, phenomena such as leading to collapse since acquisition probe pressure is excessive is also easy, eventually leads to number
According to acquisition failure or error.In consideration of it, there is an urgent need to a kind of more accurate and be resistant to the number of huge data acquisition pressure
According to acquisition solution, demand is acquired to be applicable in the data being increasingly urgent to.
Summary of the invention
In view of the above problems, the embodiment of the present invention provides a kind of collecting method and system, is adopted with solving available data
The problem of inaccuracy and unbearable huge data acquire pressure in collection scheme.
The embodiment of the present invention adopts the technical scheme that
One embodiment of the invention provides a kind of collecting method, which comprises
For each network element, acquisition probe pond is set, service is monitored in each self-starting first of each acquisition probe in acquisition probe pond,
The connection status with data processing centre is periodically monitored, and service is monitored in heart starting second in data handling, is periodically supervised
Listen the connection status with each acquisition probe；
When data acquisition session need to be carried out, the data processing centre for being in existing state is obtained according to snoop results, and
Network element to be collected corresponds to the acquisition probe that existing state is in acquisition probe pond, utilizes the Data processing for being in existing state
The heart and acquisition probe carry out data acquisition session.
Obtaining the acquisition probe in existing state in the corresponding acquisition pond of network element to be collected according to snoop results is multiple, institute
State method further include:
The load state for assessing multiple acquisition probes in existing state, determines preferred acquisition according to load balancing mode
Probe；
The data processing centre and acquisition probe using in existing state carries out data acquisition session specifically:
Data acquisition session is carried out using the data processing centre and the preferred acquisition probe that are in existing state.
The load state of the multiple acquisition probes in existing state of assessment, determines preferred according to load balancing mode
Acquisition probe includes:
For each acquisition probe, calculate current CPU usage, current memory utilization rate and storage allocation space weighting it
With obtain the score value of the load state of each acquisition probe；
Using the highest acquisition probe of score value as preferred acquisition probe.
The method also includes:
After a data acquisition operations, retain the acquisition connection of this data acquisition；The acquisition is connected as using
Network between data processing centre, acquisition probe and this data acquisition network member for executing the foundation of this data acquisition operations
Connection relationship；
If new data acquisition operations belong to same acquisition tasks with data acquisition operations before, in the acquisition of reservation
Whether the acquisition of the data acquisition before judging in connection connects within term of validity, if directlying adopt within term of validity
The acquisition connection of data acquisition before carries out the new data acquisition operations, if not executing the basis in term of validity
Snoop results obtain the data processing centre for being in existing state and network element to be collected corresponds in acquisition probe pond in survival shape
The acquisition probe of state carries out data acquisition session using the data processing centre and acquisition probe that are in existing state.
It is described to include: for each network element setting acquisition probe pond
According to the acquisition conditions of demand of network element, the acquisition probe quantity configured needed for each network element is determined；
It include the acquisition probe pond of determining acquisition probe quantity for each network element configuration.
The method also includes: it is changed according to the acquisition demand of network element, the acquisition probe quantity of corresponding change network element；
When the acquisition probe in acquisition probe pond is newly-increased or restarting, increases newly or the acquisition probe of restarting starts
One first monitors service, sends connection request to data processing centre, after the data processing centre is connected to connection request, starting
It monitors and services with the second of the acquisition probe, periodically monitor the connection status with the acquisition probe；
When the acquisition probe in acquisition probe pond is reduced, data processing centre to the acquisition probe of the reduction monitor less than
When, then it closes and monitors service to the second of the acquisition probe of the reduction.
In addition, the embodiment of the invention also provides a kind of data collection system, the system comprises:
Setting and monitoring module, for acquisition probe pond to be arranged for each network element, each acquisition probe in acquisition probe pond is each
Service is monitored in self-starting first, is periodically monitored and the connection status of data processing centre, and heart starting the in data handling
Two monitor service, periodically monitor the connection status with each acquisition probe；
Data acquisition module, for being obtained according to snoop results and being in existing state when data acquisition session need to be carried out
Data processing centre and network element to be collected correspond to the acquisition probe that existing state is in acquisition probe pond, using in depositing
The data processing centre of state living and acquisition probe carry out data acquisition session.
The data acquisition module further include:
Load evaluation unit is in existing state for obtaining in the corresponding acquisition pond of network element to be collected according to snoop results
When acquisition probe is multiple, the load state of multiple acquisition probes in existing state is assessed, it is true according to load balancing mode
Determine preferred acquisition probe；
The load evaluation unit includes:
Score subelement, for calculating current CPU usage, current memory utilization rate and distribution for each acquisition probe
The sum of weighting of memory headroom obtains the score value of the load state of each acquisition probe；
Subelement is determined, for using the highest acquisition probe of score value as preferred acquisition probe.
The data acquisition module further include:
Storage unit is connected, for after a data acquisition operations, retaining the acquisition connection of this data acquisition；
The acquisition is connected as adopting for data processing centre, acquisition probe and this data for executing the foundation of this data acquisition operations
Collect the network connection relation between network element；
Judging unit, if the acquisition of data before for new data acquisition operations and the connection storage unit to save is grasped
Work belongs to same acquisition tasks, then whether the acquisition of the data acquisition before judging in the acquisition connection of reservation connects effective
In time limit, if the acquisition connection of the data acquisition before directlying adopt carries out the new data acquisition behaviour within term of validity
Make, if not executing the data processing centre for obtaining according to snoop results and being in existing state in term of validity, and wait adopt
Collection network element correspond in acquisition probe pond be in existing state acquisition probe, using be in existing state data processing centre and
Acquisition probe carries out data acquisition session.
The setting and monitoring module include:
Acquisition probe pond determination unit determines that is configured needed for each network element adopts for the acquisition conditions of demand according to network element
Collect number of probes；
Acquisition probe pond setting unit, for including the acquisition probe of determining acquisition probe quantity for each network element configuration
Pond；
Monitoring unit is monitored service for each self-starting first of each acquisition probe in acquisition probe pond, is periodically monitored
With the connection status of data processing centre, and in data handling the heart starting second monitor service, periodically monitor with each acquisition
The connection status of probe；
The setting and monitoring module further include:
Changing unit, for being changed according to the acquisition demand of network element, the acquisition probe quantity of corresponding change network element；
Adding unit, for increasing newly or restarting when the acquisition probe in acquisition probe pond is newly-increased or restarting
Acquisition probe starting one first monitor service, send connection request to data processing centre, the data processing centre is connected to
After connection request, starting is monitored with the second of the acquisition probe and is serviced, and periodically monitors the connection status with the acquisition probe；
Unit is deleted, for when the acquisition probe in acquisition probe pond is reduced, the reduction to be adopted by data processing centre
Collect probe to monitor not then, then closes and monitor service to the second of the acquisition probe of the reduction.
Acquisition probe pond, acquisition is arranged for each network element in a kind of collecting method provided in an embodiment of the present invention and system
Service is monitored in each self-starting first of each acquisition probe in probe cell, periodically monitors the connection status with data processing centre,
And service is monitored in heart starting second in data handling, periodically monitors the connection status with each acquisition probe；When need to be counted
When according to acquisition tasks, data processing centre and the corresponding acquisition of network element to be collected in existing state are obtained according to snoop results
It is in the acquisition probe of existing state in probe cell, is counted using the data processing centre and acquisition probe that are in existing state
According to acquisition tasks.As it can be seen that the embodiment of the present invention is monitored by dynamic, it is ensured that data processing centre and acquisition probe disconnect
When, look for data processing centre and the acquisition probe of survival dynamically to carry out corresponding acquisition process, it will not be because of acquisition probe
Or data processing centre causes entire data acquisition session not execute correctly because certain reason stops.
It further,, can be in batch capture multiple devices, using more by load balancing in the embodiment of the present invention
A acquisition probe dynamic is acquired operation, and more efficiency can be improved compared with the single acquisition probe of the prior art.Also,
Compared with the prior art for single acquisition probe mechanism, it can prevent because the reasons such as network, lead to adopting for device configuration occur
Collection probe is in unusable state, and Collecting operation is caused to stop.
Detailed description of the invention
A kind of Fig. 1 collecting method flow chart provided in an embodiment of the present invention；
Fig. 2 is data acquisition network structural schematic diagram in the embodiment of the present invention；
Fig. 3 is a kind of data collection system structural block diagram provided in an embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention
Formula is described in further detail.
Referring to Fig. 1, the embodiment of the present invention provides a kind of method of data acquisition, mainly comprises the following specific steps that:
S101: being arranged acquisition probe pond for each network element, and each self-starting first of each acquisition probe in acquisition probe pond is monitored
The connection status with data processing centre is periodically monitored in service, and service, period are monitored in heart starting second in data handling
Property monitor and the connection status of each acquisition probe.
It should be noted that the embodiment of the present invention can be, but not limited to be applied to the progress data acquisition of RIM system.
RMI (Remote Method Invocation, remote method call) is realized in JDK1.1 with Java, it
Greatly strengthen the ability of Java development DDB application.RMI support is stored between the program level object in different address space
It communicates with each other, realizes the seamless far call between remote object.
It is shown in Figure 2, for acquisition probe pond schematic diagram is arranged according to step of the present invention.Here, network element is with monitored number
According to library, be managed host, other network equipments etc. for illustrated.
In order to keep the connection of acquisition probe and data processing centre, acquisition probe end and data processing centre end are opened respectively
Monitoring is moved, at regular intervals, acquisition probe or data processing centre can go dynamic searching that can currently interconnect therewith
Data processing centre or acquisition probe, after saving connection, acquisition probe pond can be used to carry out dynamically using different
Acquisition probe be acquired to be directed to different resources, it is ensured that the correct operating of data processing centre and acquisition probe,
It prevents from causing Joint failure due to network etc., task work is still carried out using the data processing centre of failure or acquisition probe
Industry causes mistake.
A kind of concrete implementation mode may is that
1, acquisition probe end distributes a fritter memory to save all data processing centres being associated.
2, data processing centre end distributes a fritter memory to save all acquisition probes being associated.
3, acquisition probe on startup, can go to initiate the connection request with data processing centre, request is by being then stored in
In the memory oneself distributed.Also, data processing centre is on startup, can go to initiate the connection request with acquisition probe, ask
It asks by being then stored in the memory oneself distributed.
4, at regular intervals, whether the monitor of data processing centre will go the acquisition probe saved in lookup memory
It can use, likewise, the monitor of acquisition probe will go to search whether the data processing centre saved in memory can be used, it is ensured that number
All it is that correctly can be used according to acquisition service, avoids mistake.In practical application, periodical monitoring can be realized using heartbeat mode.
That is, the new mechanism proposed in the embodiment of the present invention can guarantee already existing data processing centre and
Acquisition probe be it is available, not available service each other can be deleted other side from the memory of oneself by monitoring mechanism, only
Using existing for existing, connection, guarantee the correct operating of data processing centre and acquisition probe.
Preferably, described include: for each network element setting acquisition probe pond
According to the acquisition conditions of demand of network element, the acquisition probe quantity configured needed for each network element is determined；
It include the acquisition probe pond of determining acquisition probe quantity for each network element configuration.
That is, in the step of acquisition probe pond is set for each network element, preferably according to the acquisition demand of each network element
Situation is configured, in system, the data acquisition demand of some network elements is seldom, that is, the data acquisition operations in the network element
Relatively fewer, the data acquisition demand of some network elements is but very big, i.e., the data acquisition operations in the network element are relatively frequent, then right
It is relatively fewer for acquisition probe quantity included in the acquisition probe pond of its setting in the few network element of data acquisition demand, it is right
It is relatively more for acquisition probe quantity included in the acquisition probe pond of its setting in the network element more than data acquisition demand.
Further, the embodiment of the present invention the method also includes: changed according to the acquisition demand of network element, corresponding change net
The acquisition probe quantity of member.
Specifically, when the acquisition probe in acquisition probe pond is reduced, acquisition probe of the data processing centre to the reduction
It monitors then, then not close and the second monitoring of the acquisition probe of the reduction is serviced.It is i.e. subsequent to no longer need to be monitored it.It is real
In the application of border, it can be and delete the acquisition probe of the reduction in oneself memory.
When the acquisition probe in acquisition probe pond is newly-increased or restarting, increases newly or the acquisition probe of restarting starts
One first monitors service, sends connection request to data processing centre, after the data processing centre is connected to connection request, starting
It monitors and services with the second of the acquisition probe, periodically monitor the connection status with the acquisition probe.It is i.e. subsequent will periodically it is right
It is monitored.In practical application, which is added in oneself memory.
S102: when data acquisition session need to be carried out, the Data processing for being in existing state is obtained according to snoop results
The heart and network element to be collected correspond to the acquisition probe that existing state is in acquisition probe pond, utilize the data for being in existing state
Processing center and acquisition probe carry out data acquisition session.
Preferably, obtaining the acquisition probe in existing state in the corresponding acquisition pond of network element to be collected according to snoop results is
It is multiple, the method also includes:
The load state for assessing multiple acquisition probes in existing state, determines preferred acquisition according to load balancing mode
Probe.
Correspondingly, the data processing centre and acquisition probe using in existing state carries out data acquisition session tool
Body are as follows:
Data acquisition session is carried out using the data processing centre and the preferred acquisition probe that are in existing state.
As a preferred embodiment, the load state of the multiple acquisition probes in existing state of assessment, according to negative
It carries balanced way and determines that preferred acquisition probe includes:
For each acquisition probe, calculate current CPU usage, current memory utilization rate and storage allocation space weighting it
With obtain the score value of the load state of each acquisition probe；
Using the highest acquisition probe of score value as preferred acquisition probe.
That is, using the load-balancing mechanism in acquisition probe pond, it is ensured that in the process of implementation, can be dynamic
Distribution acquisition probe acquires data, and the acquisition probe that selects pressure small carries out data acquisition, it is ensured that data it is correct
Acquisition.
A kind of specific load state appraisal procedure calculates current CPU usage, current memory for each acquisition probe
The sum of utilization rate and the weighting in storage allocation space, the score value for obtaining the load state of each acquisition probe may is that
The load state score value of acquisition probe=
In current CPU usage * utilization rate weight coefficient+current memory utilization rate * memory usage weight coefficient+distribution
Deposit space * memory headroom weight coefficient
That is, the embodiment of the present invention proposes a kind of specific load state assessment algorithm, i.e., in due course system scoring is calculated
Method: the performance indicator that the algorithm is mainly used to describe the acquisition probe that current server is disposed is shown, is taken by timing acquisition
Be engaged in device CPU and currently used rate (CP that currently scores is obtained according to master frequency and current CPU usage), memory with it is currently used
Rate (obtains currently available memory and obtains current scoring MP), and the memory that JVM is distributed when starting services (obtains current JVM distribution
Stack memory, obtain JVM configuration scoring JP) etc. information, be calculated each acquisition probe index comprehensive assessment numerical value.
In practical application, in order to further increase the accuracy of assessment, it is current that load state assessment algorithm can be judgement
Whether CPU usage CP, current memory utilization rate MP and tri- values of storage allocation space JP differ greatly, for example utilize above-mentioned
Maximum value in three values judges whether ratio is more than preset value (assuming that can decide whether to be greater than 120%) divided by minimum value,
If more than then illustrating to differ larger, if being less than, illustrate to be not much different.
If the maximum value in CP, MP, JP tri- values divided by minimum value, is more than as a result preset value, then illustrates CP, MP, JP tri-
A value difference is larger, in order to further decrease error, is assessed using the smallest value in tri- values of CP, MP, JP, it is preferred that
Utilization rate weight coefficient is set as 100%, and memory usage weight coefficient and memory headroom weight coefficient are set as 0%.
If as a result the maximum value in CP, MP, JP tri- values is less than preset value, then illustrates CP, MP, JP divided by minimum value
Three values are not much different, and corresponding weight coefficient can be arranged according to influence degree of three values to load state, it is preferred that benefit
It is set as 40% with rate weight coefficient, memory usage weight coefficient is set as 40%, and memory headroom weight coefficient is set as
20%.
For example: it is directed to each probe, it is maximum divided by the smallest in CP, MP, JP tri- values, if more than 120%,
Then use the smallest scoring as the probe in three；If it is maximum divided by the smallest less than 120%, use CP*40%+MP*
Scoring of the 40%+JP*20% as the probe.
Load balancing uses algorithm, can be for spy that is least recently used, executing minimum task in acquisition probe
Needle carries out examination use, using dynamic allocation, guarantees to a certain extent, to guarantee that acquisition probe selected by resource is current pressure
The smallest acquisition probe.
Further, the embodiment of the present invention the method also includes:
After a data acquisition operations, retain the acquisition connection of this data acquisition；The acquisition is connected as using
Network between data processing centre, acquisition probe and this data acquisition network member for executing the foundation of this data acquisition operations
Connection relationship.
If new data acquisition operations belong to same acquisition tasks with data acquisition operations before, in the acquisition of reservation
Whether the acquisition of the data acquisition before judging in connection connects within term of validity, if directlying adopt within term of validity
The acquisition connection of data acquisition before carries out the new data acquisition operations, if not executing the basis in term of validity
Snoop results obtain the data processing centre for being in existing state and network element to be collected corresponds in acquisition probe pond in survival shape
The acquisition probe of state carries out data acquisition session using the data processing centre and acquisition probe that are in existing state.
That is, a data acquisition session executes realization by multiple data acquisition operations, as a data acquisition behaviour
After work, connection used in the data acquisition operations is saved, if subsequent data acquisition operations are acquired with data before
Operation belongs to same data acquisition session, then connects, sentence used in data acquisition operations before searching in the connection of preservation
It is disconnected if so, being not necessarily to assess the load state of each acquisition probe again, and to re-establish connection whether within term of validity,
Further increase data acquisition efficiency.If needing to reappraise the load state of each acquisition probe not in term of validity, laying equal stress on
Newly establish data acquisition connection.
Acquisition probe pond, acquisition probe pond is arranged for each network element in a kind of collecting method provided in an embodiment of the present invention
In each self-starting first of each acquisition probe monitor service, periodically monitor the connection status with data processing centre, and in number
Service is monitored according to processing center starting second, periodically monitors the connection status with each acquisition probe；When data acquisition need to be carried out
When task, the data processing centre for being in existing state is obtained according to snoop results and network element to be collected corresponds to acquisition probe pond
In be in existing state acquisition probe, utilize in existing state data processing centre and acquisition probe carry out data acquisition
Task.As it can be seen that the embodiment of the present invention is monitored by dynamic, it is ensured that data processing centre and acquisition probe are moved when disconnecting
State looks for data processing centre and the acquisition probe of survival to carry out corresponding acquisition process, will not be because of acquisition probe or data
Processing center is because certain reason stops, and entire data acquisition session is caused not execute correctly.
It further,, can be in batch capture multiple devices, using more by load balancing in the embodiment of the present invention
A acquisition probe dynamic is acquired operation, and more efficiency can be improved compared with the single acquisition probe of the prior art.Also,
Compared with the prior art for single acquisition probe mechanism, it can prevent because the reasons such as network, lead to adopting for device configuration occur
Collection probe is in unusable state, and Collecting operation is caused to stop.
It referring to Fig. 3, and combines shown in above-mentioned Fig. 2, the embodiment of the present invention provides a kind of data collection system, the system packet
It includes:
Setting and monitoring module 301, for acquisition probe pond to be arranged for each network element, each acquisition probe in acquisition probe pond
Service is monitored in each self-starting first, periodically monitors the connection status with data processing centre, and the heart starts in data handling
Second monitors service, periodically monitors the connection status with each acquisition probe.
Data acquisition module 302, for being obtained in survival shape according to snoop results when data acquisition session need to be carried out
The data processing centre of state and network element to be collected correspond to the acquisition probe that existing state is in acquisition probe pond, using being in
The data processing centre of existing state and acquisition probe carry out data acquisition session.
Preferably, the data acquisition module further include:
Load evaluation unit is in existing state for obtaining in the corresponding acquisition pond of network element to be collected according to snoop results
When acquisition probe is multiple, the load state of multiple acquisition probes in existing state is assessed, it is true according to load balancing mode
Determine preferred acquisition probe.
Specifically, the load evaluation unit includes:
Score subelement, for calculating current CPU usage, current memory utilization rate and distribution for each acquisition probe
The sum of weighting of memory headroom obtains the score value of the load state of each acquisition probe；
Subelement is determined, for using the highest acquisition probe of score value as preferred acquisition probe.
Further, the data acquisition module further include:
Storage unit is connected, for after a data acquisition operations, retaining the acquisition connection of this data acquisition；
The acquisition is connected as adopting for data processing centre, acquisition probe and this data for executing the foundation of this data acquisition operations
Collect the network connection relation between network element；
Judging unit, if the acquisition of data before for new data acquisition operations and the connection storage unit to save is grasped
Work belongs to same acquisition tasks, then whether the acquisition of the data acquisition before judging in the acquisition connection of reservation connects effective
In time limit, if the acquisition connection of the data acquisition before directlying adopt carries out the new data acquisition behaviour within term of validity
Make, if not executing the data processing centre for obtaining according to snoop results and being in existing state in term of validity, and wait adopt
Collection network element correspond in acquisition probe pond be in existing state acquisition probe, using be in existing state data processing centre and
Acquisition probe carries out data acquisition session.
As specifically, the setting and monitoring module include:
Acquisition probe pond determination unit determines that is configured needed for each network element adopts for the acquisition conditions of demand according to network element
Collect number of probes；
Acquisition probe pond setting unit, for including the acquisition probe of determining acquisition probe quantity for each network element configuration
Pond；
Monitoring unit is monitored service for each self-starting first of each acquisition probe in acquisition probe pond, is periodically monitored
With the connection status of data processing centre, and in data handling the heart starting second monitor service, periodically monitor with each acquisition
The connection status of probe.
Preferably, the setting and monitoring module further include:
Changing unit, for being changed according to the acquisition demand of network element, the acquisition probe quantity of corresponding change network element；
Adding unit, for increasing newly or restarting when the acquisition probe in acquisition probe pond is newly-increased or restarting
Acquisition probe starting one first monitor service, send connection request to data processing centre, the data processing centre is connected to
After connection request, starting is monitored with the second of the acquisition probe and is serviced, and periodically monitors the connection status with the acquisition probe；
Unit is deleted, for when the acquisition probe in acquisition probe pond is reduced, the reduction to be adopted by data processing centre
Collect probe to monitor not then, then closes and monitor service to the second of the acquisition probe of the reduction.
It should be noted that the working principle and treatment process of modules or unit in present system embodiment
It may refer to the associated description in embodiment of the method shown in above-mentioned Fig. 1-Fig. 2, details are not described herein again.
Acquisition probe pond, acquisition probe pond is arranged for each network element in a kind of data collection system provided in an embodiment of the present invention
In each self-starting first of each acquisition probe monitor service, periodically monitor the connection status with data processing centre, and in number
Service is monitored according to processing center starting second, periodically monitors the connection status with each acquisition probe；When data acquisition need to be carried out
When task, the data processing centre for being in existing state is obtained according to snoop results and network element to be collected corresponds to acquisition probe pond
In be in existing state acquisition probe, utilize in existing state data processing centre and acquisition probe carry out data acquisition
Task.As it can be seen that the embodiment of the present invention is monitored by dynamic, it is ensured that data processing centre and acquisition probe are moved when disconnecting
State looks for data processing centre and the acquisition probe of survival to carry out corresponding acquisition process, will not be because of acquisition probe or data
Processing center is because certain reason stops, and entire data acquisition session is caused not execute correctly.
It further,, can be in batch capture multiple devices, using more by load balancing in the embodiment of the present invention
A acquisition probe dynamic is acquired operation, and more efficiency can be improved compared with the single acquisition probe of the prior art.Also,
Compared with the prior art for single acquisition probe mechanism, it can prevent because the reasons such as network, lead to adopting for device configuration occur
Collection probe is in unusable state, and Collecting operation is caused to stop.
For the ease of clearly describing the technical solution of the embodiment of the present invention, in the embodiment of invention, use " first ",
Printed words such as " second " distinguish function and the essentially identical identical entry of effect or similar item, and those skilled in the art can manage
The printed words such as solution " first ", " second " are not defined quantity and execution order.
It will appreciated by the skilled person that implement the method for the above embodiments be can be with
Relevant hardware is instructed to complete by program, the program can be stored in a computer readable storage medium,
The program when being executed, includes the following steps: (the step of method), the storage medium, such as: ROM/RAM, magnetic disk, CD
Deng.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (7)
1. a kind of collecting method, which is characterized in that the described method includes:
Acquisition probe pond is set for each network element, service, period are monitored in each self-starting first of each acquisition probe in acquisition probe pond
Property monitor with the connection status of data processing centre, and service is monitored in heart starting second in data handling, periodically monitor with
The connection status of each acquisition probe；
When data acquisition session need to be carried out, the data processing centre for being in existing state is obtained according to snoop results, and wait adopt
Collection network element correspond in acquisition probe pond be in existing state acquisition probe, using be in existing state data processing centre and
Acquisition probe carries out data acquisition session；
Obtaining the acquisition probe in existing state in the corresponding acquisition pond of network element to be collected according to snoop results is multiple, the side
Method further include:
The load state for assessing multiple acquisition probes in existing state determines that preferred acquisition is visited according to load balancing mode
Needle；
The data processing centre and acquisition probe using in existing state carries out data acquisition session specifically:
Data acquisition session is carried out using the data processing centre and the preferred acquisition probe that are in existing state；
The method also includes:
After a data acquisition operations, retain the acquisition connection of this data acquisition；The acquisition is connected as holding
The network connection between data processing centre, acquisition probe and this data acquisition network member that this data acquisition operations of row are established
Relationship；
If new data acquisition operations belong to same acquisition tasks with data acquisition operations before, connected in the acquisition of reservation
Whether the acquisition of the data acquisition before middle judgement connects within term of validity, if within term of validity, before directlying adopt
Data acquisition acquisition connect carry out the new data acquisition operations, if not executed described according to monitoring in term of validity
As a result it obtains the data processing centre for being in existing state and network element to be collected corresponds to and is in existing state in acquisition probe pond
Acquisition probe carries out data acquisition session using the data processing centre and acquisition probe that are in existing state.
2. collecting method according to claim 1, which is characterized in that multiple the adopting in existing state of assessment
The load state for collecting probe, determines that preferred acquisition probe includes: according to load balancing mode
For each acquisition probe, the sum of current CPU usage, current memory utilization rate and the weighting in storage allocation space are calculated,
Obtain the score value of the load state of each acquisition probe；
Using the highest acquisition probe of score value as preferred acquisition probe.
3. collecting method according to claim 1, which is characterized in that described that acquisition probe Chi Bao is arranged for each network element
It includes:
According to the acquisition conditions of demand of network element, the acquisition probe quantity configured needed for each network element is determined；
It include the acquisition probe pond of determining acquisition probe quantity for each network element configuration.
4. collecting method according to claim 3, which is characterized in that the method also includes: according to adopting for network element
The change of collection demand, the acquisition probe quantity of corresponding change network element；
When the acquisition probe in acquisition probe pond is newly-increased or restarting, increase newly or the acquisition probe of restarting starting 1 the
One monitoring service, sends connection request to data processing centre, after the data processing centre is connected to connection request, starts and is somebody's turn to do
The second of acquisition probe monitors service, periodically monitors the connection status with the acquisition probe；
When the acquisition probe in acquisition probe pond is reduced, the acquisition probe of the reduction is monitored not then by data processing centre,
It then closes to monitor the second of the acquisition probe of the reduction and service.
5. a kind of data collection system, which is characterized in that the system comprises:
Setting and monitoring module, for acquisition probe pond to be arranged for each network element, each acquisition probe in acquisition probe pond is respectively opened
Dynamic first monitors service, periodically monitors the connection status with data processing centre, and the second prison of heart starting in data handling
Service is listened, the connection status with each acquisition probe is periodically monitored；
Data acquisition module, for obtaining the number for being in existing state according to snoop results when data acquisition session need to be carried out
The acquisition probe that existing state is in acquisition probe pond is corresponded to according to processing center and network element to be collected, using in survival shape
The data processing centre of state and acquisition probe carry out data acquisition session；
The data acquisition module further include: load evaluation unit is adopted for obtaining network element correspondence to be collected according to snoop results
When the acquisition probe in existing state is multiple in collection pond, the load shape of multiple acquisition probes in existing state is assessed
Condition determines preferred acquisition probe according to load balancing mode；
The data acquisition module further include:
Storage unit is connected, for after a data acquisition operations, retaining the acquisition connection of this data acquisition；It is described
Acquisition is connected as data processing centre, acquisition probe and this data acquisition network for executing the foundation of this data acquisition operations
Network connection relation between member；
Judging unit, if the data acquisition operations category before saved for new data acquisition operations and the connection storage unit
In same acquisition tasks, then whether the acquisition of the data acquisition before judging in the acquisition connection of reservation connects in term of validity
It is interior, if the acquisition connection of the data acquisition before directlying adopt carries out the new data acquisition operations within term of validity, if
Not in term of validity, then the data processing centre according to snoop results acquisition in existing state and net to be collected are executed
It is in the acquisition probe of existing state in the corresponding acquisition probe pond of member, utilizes the data processing centre and acquisition for being in existing state
Probe carries out data acquisition session.
6. data collection system according to claim 5, which is characterized in that the load evaluation unit includes:
Score subelement, for calculating current CPU usage, current memory utilization rate and storage allocation for each acquisition probe
The sum of the weighting in space obtains the score value of the load state of each acquisition probe；
Subelement is determined, for using the highest acquisition probe of score value as preferred acquisition probe.
7. data collection system according to claim 5, which is characterized in that the setting and monitoring module include:
Acquisition probe pond determination unit determines that the acquisition configured needed for each network element is visited for the acquisition conditions of demand according to network element
Needle quantity；
Acquisition probe pond setting unit, for including the acquisition probe pond of determining acquisition probe quantity for each network element configuration；
Monitoring unit is monitored service for each self-starting first of each acquisition probe in acquisition probe pond, is periodically monitored and number
According to the connection status of processing center, and in data handling, service is monitored in heart starting second, is periodically monitored and each acquisition probe
Connection status；
The setting and monitoring module further include:
Changing unit, for being changed according to the acquisition demand of network element, the acquisition probe quantity of corresponding change network element；
Adding unit, it is newly-increased or restarting to adopt for when the acquisition probe in acquisition probe pond is newly-increased or restarting
Collect probe starting one first and monitor service, sends connection request to data processing centre, the data processing centre is connected to connection
After request, starting is monitored with the second of the acquisition probe and is serviced, and periodically monitors the connection status with the acquisition probe；
Unit is deleted, for when the acquisition probe in acquisition probe pond is reduced, the acquisition of the reduction to be visited by data processing centre
Needle is monitored not then, then closes and monitor service to the second of the acquisition probe of the reduction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410280346.1A CN105281967B (en) | 2014-06-20 | 2014-06-20 | A kind of collecting method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410280346.1A CN105281967B (en) | 2014-06-20 | 2014-06-20 | A kind of collecting method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105281967A CN105281967A (en) | 2016-01-27 |
| CN105281967B true CN105281967B (en) | 2019-06-28 |
Family
ID=55150337
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410280346.1A Active CN105281967B (en) | 2014-06-20 | 2014-06-20 | A kind of collecting method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105281967B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112311878A (en) * | 2020-10-30 | 2021-02-02 | 杭州迪普科技股份有限公司 | Centralized control probe disconnection reconnection method, device and system |
| CN114007232A (en) * | 2020-12-14 | 2022-02-01 | 北京八分量信息科技有限公司 | Keep-alive mechanism realized by optimizing node communication |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6278694B1 (en) * | 1999-04-16 | 2001-08-21 | Concord Communications Inc. | Collecting and reporting monitoring data from remote network probes |
| GB2371442A (en) * | 2000-11-01 | 2002-07-24 | Agilent Technologies Inc | Monitoring communication network configuration by probing and characterising network data streams |
| CN1960274A (en) * | 2005-11-01 | 2007-05-09 | 中兴通讯股份有限公司 | Method for dynamic real time capturing logic commands input from UNIX terminal user |
| CN101201782A (en) * | 2006-12-01 | 2008-06-18 | 株式会社日立制作所 | Debug information collection method and debug information collection system |
| CN102143022A (en) * | 2011-03-16 | 2011-08-03 | 北京邮电大学 | Cloud measurement device and method for IP network |
| CN102801587A (en) * | 2012-08-29 | 2012-11-28 | 北京邮电大学 | Large-scale network-oriented virtualized monitoring system and dynamic monitoring method thereof |
-
2014
- 2014-06-20 CN CN201410280346.1A patent/CN105281967B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6278694B1 (en) * | 1999-04-16 | 2001-08-21 | Concord Communications Inc. | Collecting and reporting monitoring data from remote network probes |
| GB2371442A (en) * | 2000-11-01 | 2002-07-24 | Agilent Technologies Inc | Monitoring communication network configuration by probing and characterising network data streams |
| CN1960274A (en) * | 2005-11-01 | 2007-05-09 | 中兴通讯股份有限公司 | Method for dynamic real time capturing logic commands input from UNIX terminal user |
| CN101201782A (en) * | 2006-12-01 | 2008-06-18 | 株式会社日立制作所 | Debug information collection method and debug information collection system |
| CN102143022A (en) * | 2011-03-16 | 2011-08-03 | 北京邮电大学 | Cloud measurement device and method for IP network |
| CN102801587A (en) * | 2012-08-29 | 2012-11-28 | 北京邮电大学 | Large-scale network-oriented virtualized monitoring system and dynamic monitoring method thereof |
Non-Patent Citations (1)
| Title |
|---|
| 《IPTV性能测量探针的设计与实现》;姜志;《中国优秀硕士学位论文全文数据库（信息科技辑）》;20110315;第2011卷(第03期);第I136-1528页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105281967A (en) | 2016-01-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102693177B (en) | Fault diagnosing and processing methods of virtual machine as well as device and system thereof | |
| CN108205541B (en) | Method and device for scheduling distributed web crawler tasks | |
| US7797572B2 (en) | Computer system management method, management server, computer system, and program | |
| US10348809B2 (en) | Naming of distributed business transactions | |
| CN106375420B (en) | Server cluster intelligent monitoring system and method based on load balancing | |
| CN105357038B (en) | Monitor the method and system of cluster virtual machine | |
| CN105468450B (en) | Method for scheduling task and system | |
| CN103441906B (en) | Based on from the proxy caching cluster abnormality detection system of host computer | |
| CN106452818A (en) | Resource scheduling method and resource scheduling system | |
| CN104102543B (en) | The method and apparatus of adjustment of load in a kind of cloud computing environment | |
| EP2240858B1 (en) | Method for using dynamically scheduled synthetic transactions to monitor performance and availability of e-business systems | |
| CN111694515B (en) | Zone writing distribution method and system based on ZNS solid state disk | |
| CN110825544A (en) | Computing node, failure detection method thereof and cloud data processing system | |
| US20130198370A1 (en) | Method for visualizing server reliability, computer system, and management server | |
| CN112231075A (en) | Server cluster load balancing control method and system based on cloud service | |
| CN108633311A (en) | A kind of method, apparatus and control node of the con current control based on call chain | |
| CN106789189B (en) | SOA service real-time monitoring reporting and performance measurement system | |
| CN111124609B (en) | Data acquisition method and device, data acquisition equipment and storage medium | |
| US20100042996A1 (en) | Utilization management | |
| GB2613125A (en) | Storage level load balancing | |
| CN105281967B (en) | A kind of collecting method and system | |
| US8973000B2 (en) | Determining multiprogramming levels | |
| CN107844406A (en) | Method for detecting abnormality and system, service terminal, the memory of distributed system | |
| CN111738837A (en) | Credit card limit processing method and device | |
| CN117632897A (en) | Dynamic capacity expansion and contraction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 818, 8 / F, 34 Haidian Street, Haidian District, Beijing 100080 Patentee after: BEIJING ULTRAPOWER SOFTWARE Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building 6 storey block A Room 601 Patentee before: BEIJING ULTRAPOWER SOFTWARE Co.,Ltd. |
|
| CP02 | Change in the address of a patent holder |