CN115811457A - Integrated data acquisition method for multi-source data fusion - Google Patents
Integrated data acquisition method for multi-source data fusion Download PDFInfo
- Publication number
- CN115811457A CN115811457A CN202211530609.0A CN202211530609A CN115811457A CN 115811457 A CN115811457 A CN 115811457A CN 202211530609 A CN202211530609 A CN 202211530609A CN 115811457 A CN115811457 A CN 115811457A
- Authority
- CN
- China
- Prior art keywords
- data
- 7xops
- task
- agent
- monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention belongs to the technical field of multi-source data fusion, and discloses an integrated data acquisition method for multi-source data fusion, wherein a 7xOPS Agent is automatically and manually registered: s1: realizing global initialization of a configuration file, global initialization of a print log, configuration of an agent IP pool and system initialization; s2: requesting 7xOPS Server to distribute Token, automatically requesting 7xOPS Server to acquire Token every 5 minutes, and starting in a coroutine mode; the invention provides an integrated data acquisition method for multi-source data fusion, which can realize the operations of installation, upgrade, uninstallation, editing and the like by managing a 7xOPS Agent through a 7xOPS Server, and the task types comprise tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring, UCPE monitoring and the like. The 7xOPS Server realizes the operations of starting, stopping, editing, deleting and the like of tasks, realizes the data acquisition of different task types through the 7xOPS Agent, and realizes the storage of the acquired data through the 7xOPS Receiver.
Description
Technical Field
The invention belongs to the technical field of multi-source data fusion, and particularly relates to an integrated data acquisition method for multi-source data fusion.
Background
With the popularization of broadband networks and the increase of the demands of management departments for management level improvement, the demands of management departments such as urban public security, traffic, finance, environmental protection, electric power, medical treatment, education and the like on large networking security monitoring platforms in urban areas begin to increase greatly in two years, with the maturity of network transmission technology and the internet, great breakthrough is made on the realization of customer demands of emerging industries, large-scale automatic operation and maintenance monitoring systems are rapidly built all over the country, operating systems supported by mainstream of the existing common collectors are Windows, various Linux, unix and the like, research and development are carried out on specific operating systems, each operating system has an independent development environment, the research and development cost is high, codes are difficult to maintain and the like.
Disclosure of Invention
In order to solve the problems in the background art, the invention provides an integrated data acquisition method for multi-source data fusion, wherein 7xOPS agents are managed through a 7xOPS Server, so that the operations of installation, upgrading, unloading and the like can be realized, and the task types comprise tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring, UCPE monitoring and the like. The 7xOPS Server realizes the operations of task deletion, editing, starting, stopping and the like, the 7xOPS Agent realizes the unified entry management and data acquisition of different task types, and the 7xOPS Receiver realizes the storage of the acquired data.
In order to achieve the purpose, the invention provides the following technical scheme: in the integrated data acquisition method of multi-source data fusion, the 7xOPS Agent is automatically and manually registered;
s1: realizing global initialization of a configuration file, global initialization of a print log, configuration of an agent IP pool and system initialization;
s2: requesting the 7xOPS Server to acquire the Token, automatically requesting the 7xOPS Server to acquire the Token every 5 minutes, and starting in a coroutine mode;
the Token generation rule is randomly generated according to a user name and a password, an assignment operation is carried out after a returned Token is obtained, and a 7xOPS Agent is assigned to an object of global configuration;
s3: performing registration verification on the acquired Token in a 7xOPS Server, and judging whether the 7xOPS Agent is automatically/manually deployed;
s31: automatically deploying a request 7xOPS Server, updating the management state of the 7xOPS Agent, returning a request result, if the request result is successful, updating the global configuration, and if the request result is failed, exiting the current function, and re-initiating the request;
s32: the manual deployment requests the 7xOPS Server, newly deployed 7xOPS agents are registered, if the registration is successful, the global configuration is updated, if the registration is failed, the program is quitted, and the process is ended;
s4: the 7 xpps Agent registration is complete.
Preferably, the 7 xpos Agent deploys automatically:
s1: creating a new Agent in the 7xOPS Agent management list, wherein the new Agent needs to install an address IP, a root account number and a password, an installation path, a download link and other information;
s2: the 7xOPS Server starts SSH to establish a link, uploads a shell script after the link is established successfully, executes the shell script, jumps to a specified path, downloads an installation package file, and starts after the installation package file is downloaded successfully;
s3: the 7xOPS Agent is successfully started, the registration is completed in the 7xOPS Server, the 7xOPS Agent can regularly send the Agent state for updating, and an installation log is attached to the Agent state for sending to the Mq;
s4: and the 7xOPS Server receives the state updating message from the Mq, modifies the state of the 7xOPS Agent created in the 7xOPS Agent management list into the running state, and saves the installation log.
Preferably, the 7 xpos Agent automatically upgrades and uninstalls:
s1: clicking and upgrading in the 7xOPS Agent management list, issuing a task to a corresponding Mq queue by the 7xOPS Server, and monitoring the Mq task issued by the 7xOPS Server by the 7xOPS Agent;
s2: the 7xOPS Agent monitors information to be upgraded, analyzes the received information, executes a shell script, downloads an installation package through a link, and judges whether the 7xOPS Agent is started or not through a checking process after the installation is finished;
s21: in the starting process: backing up the currently running installation package, killing the current program process, running a new installation package, and sending a state message to the Mq task after the successful start;
s22: and (3) no start-up: backing up an old installation package, operating a new installation package, and sending a state message to an Mq task after the new installation package is successfully started;
s3: and the 7xOPS Server receives the status message from the Mq, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list into the running status, and saves the upgrade log.
Preferably, the monitoring task acquisition:
the 7xOPS Server realizes the management of a 7xOPS Agent monitoring task, performs the starting and stopping of various tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring, and sends a monitored information template to a corresponding Mq queue according to the task type after the task is started;
the 7xOPS Agent monitors the Mq task issued by the 7xOPS Server, monitors different data according to the type of the task, and performs data cleaning, data filtering, data integration, data queuing, data structure optimization, data compression and data transmission on the monitored data;
the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
Preferably, the performance acquisition monitoring process includes:
s1: the 7xOPS Agent task list starts a performance monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: performing Context assignment on the verified data to enable the data to be used in the whole process in the following monitoring task, installing a plug-in shell script and customizing a collection index by a user;
s3: running a performance monitoring task, and reading data by using a Linux (acquiring built-in files) and a Windows (using commands and dll files) method to acquire data such as system equipment information cpu, an internal memory, a wireless network card and the like;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize the storage of the collected data;
the SNMP acquisition process comprises the following steps:
s1: the 7xOPS Agent task list starts an SNMP monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: context assignment is carried out on the checked data, so that the data can be used in the whole process in the following monitoring task;
s3: the SNMP monitoring task is operated, basic information such as a cpu, an internal memory and the like of the equipment can be obtained by using an SNMP walk/get request method through Oid and IP as access parameters, and data monitoring can also be carried out through a Ping method;
s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
Preferably, the asset scanning procedure:
s1: the 7xOPS Agent task list starts an asset scanning task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;
s2: through the checked data, the task Id needs to be added into a global variable Map array created by the asset scanning task, so that the received task can be started/stopped subsequently;
s3: running an asset scanning task, performing different data monitoring according to data types (icmp/SNMP) in task information, and acquiring system environment basic information by using an SNMP walk/get request method through Oid and IP as access parameters or performing data monitoring through a Ping method;
s4: carrying out data cleaning and data filtering on the monitored data, and carrying out data transmission after data integration;
s5: and the data monitored by the 7xOPS Agent is sent to the 7xOPS Server and the 7xOPS Server to realize the storage of the acquired data.
Preferably, the state monitoring process:
s1: the 7xOPS Agent task list starts a state monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to start;
s2: adding the task Id into a global variable Map array created by the state monitoring task to enable the received task to be started and stopped subsequently;
s3: running a state monitoring task, monitoring data according to the IP in the task information, and acquiring data such as whether normal communication can be realized by using a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;
the sensor monitoring process comprises the following steps:
s1: the 7xOPS Agent task list starts a sensor monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to start;
s2: adding the task Id into a global variable Map array created by the sensor monitoring task to enable the received task to be started and stopped subsequently;
s3: running a sensor monitoring task, sending equipment information to a specified queue by an IOT (input/output) equipment used by a user at regular time, starting monitoring the queue by a 7xOPS (application operating system) Agent after the task is started, and reading data by a method for monitoring an exclusive queue;
s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
Preferably, the domain name detection process:
s1: the 7xOPS Agent task list starts a domain name monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;
s2: adding the task Id into a global variable Map array created by the domain name monitoring task to enable the received task to be started and stopped subsequently;
s3: running a domain name monitoring task, monitoring data according to IP in task information, and acquiring data such as packet loss, time delay, state codes and the like by using a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
Preferably, the UCPE acquisition process:
s1: the 7xOPS Agent task list starts a UCPE monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: adding the task Id into a global variable Map array created by the UCPE monitoring task to enable the received task to be started and stopped subsequently;
s3: running the UCPE monitoring task, executing the shell script to obtain 7xNetWork equipment information, and reading the data monitored by the shell script;
s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;
s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
Preferably, the 7xOPS Agent task starts and stops: the 7xOPS Server realizes the management of the 7xOPS Agent monitoring task, and can start and stop different tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring;
when the 7xOPS Server issues the task to the 7xOPS Agent through Mq, the 7xOPS Agent runs a start and stop program according to the data analyzed in the issuing.
Compared with the prior art, the invention has the following beneficial effects:
the invention provides an integrated data acquisition method for multi-source data fusion, which can realize the operations of installation, upgrade, uninstallation and the like by managing a 7xOPS Agent through a 7xOPS Server, and the task types comprise tasks of performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring, UCPE monitoring and the like. The 7xOPS Server realizes the operations of installation, upgrade, uninstallation, starting, stopping and the like of tasks, realizes unified entry management and data acquisition of different task types through the 7xOPS Agent, and realizes the storage of the acquired data through the 7xOPS Receiver.
Drawings
FIG. 1 is a schematic block diagram of the present invention;
FIG. 2 is a schematic diagram of the automatic and manual registration process of the 7xOPS Agent according to the present invention;
FIG. 3 is a schematic diagram illustrating an automatic deployment process of 7xOPS agents according to the present invention;
FIG. 4 is a schematic diagram of the automatic upgrade and uninstall process of the 7xOPS Agent of the present invention;
FIG. 5 is a schematic view of a task collection process of the present invention;
FIG. 6 is a schematic diagram of a performance collection process of the present invention;
FIG. 7 is a schematic diagram of an SNMP collection process according to the present invention;
FIG. 8 is a schematic view of an asset scanning flow of the present invention;
FIG. 9 is a schematic diagram of a status detection process according to the present invention;
FIG. 10 is a schematic diagram of a sensor detection process according to the present invention;
FIG. 11 is a schematic view of a domain name detection process according to the present invention;
FIG. 12 is a schematic view of a UCPE acquisition process according to the present invention;
FIG. 13 is a schematic diagram of the start and stop processes of a 7xOPS Agent task according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1 to 13, the present invention provides an integrated data acquisition method for multi-source data fusion, and 7xOPS Agent automatic and manual registration:
s1: realizing global initialization of configuration files, global initialization of printing logs, configuration of proxy IP pools and system initialization;
s2: requesting the 7xOPS Server to acquire the Token, automatically requesting the 7xOPS Server to acquire the Token every 5 minutes, and starting in a coroutine mode;
a Token generation rule is randomly generated according to the user name and the password, and after the returned Token is obtained, assignment operation is carried out to assign the Token to the object of global configuration;
s3: performing registration verification on the acquired Token in a 7xOPS Server, and judging whether the 7xOPS Agent is automatically/manually deployed;
s31: automatically deploying a request 7xOPS Server, updating the management state of the 7xOPS Agent, returning a request result, if the request result is successful, updating the global configuration, and if the request result is failed, exiting the current function and re-requesting;
s32: the manual deployment request 7xOPS Server registers the newly deployed 7xOPS Agent, if the registration is successful, the global configuration is updated, if the registration is failed, the program is quitted, and the process is ended;
s4:7xOPS Agent registration is completed;
as shown in fig. 3, the 7xOPS Agent deploys automatically:
s1: creating a new monitoring task in a 7xOPS Agent management list, wherein the new monitoring task needs to install an address IP, a root account number and a password, install a path, download a link and other information;
s2: the 7xOPS Server starts SSH to establish a link, uploads a shell script after the link is established successfully, executes the shell script, jumps to a specified path, downloads an installation package file, and starts after the download is successful;
s3: the 7xOPS Agent is started and succeeds, registration is completed in the 7xOPS Server, and the 7xOPS Agent sends monitoring task state update at regular time and additionally installs a log;
s4: and the 7xOPS Server receives the state updating message, modifies the state of the newly created 7xOPS Agent in the 7xOPS Agent management list into the running state, and saves the installation log.
As shown in fig. 4, the 7xOPS Agent automatically upgrades and uninstalls:
s1: clicking and starting the 7xOPS Agent management list, issuing a task to a corresponding Mq queue by the 7xOPS Server, and monitoring the Mq task issued by the 7xOPS Server by the 7xOPS Agent;
s2: the 7xOPS Agent monitors information to be upgraded, executes a shell script, downloads an installation package through a link and judges whether the 7xOPS Agent is started or not;
s21: in the starting process: backing up the currently running installation package, killing the current program process, running a new installation package, and sending a state message to the Mq task after the successful start;
s22: and (3) no start-up: backing up an old installation package, operating a new installation package, and sending a state message to an Mq task after the new installation package is successfully started;
s3: and the 7xOPS Server receives the state updating message, modifies the state of the newly created 7xOPS Agent in the 7xOPS Agent management list into the running state, and stores the upgrading/unloading log.
As shown in fig. 5, the monitoring task collects:
the 7xOPS Server realizes the management of a 7xOPS Agent monitoring task, performs the starting and stopping of various tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring, and sends a monitored information template to a corresponding Mq queue according to the task type after the task is started;
the 7xOPS Agent monitors the task issued by the 7xOPS Server, performs different data monitoring according to the type of the task, performs data cleaning, data filtering, data integration, data queuing, data structure optimization and data compression on the monitored data, and finally performs data transmission;
the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;
the 7xOPS Agent has uniformity, is suitable for monitoring most service scenes in the market, and can also create a custom template, for example: some systems only have snmp monitoring/ping monitoring, and most systems only have snmp monitoring;
the 7xOPS system Agent uniform collector integrates tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring, UCPE monitoring, template customization and the like in a project, is simple in configuration, and facilitates operation of users, uniform management and the like.
The system compatibility is suitable for the development of system environments such as Linux, windows, mac and the like, for example: due to the non-uniformity of system commands in Linux, windows and Mac, project development needs to be independently developed in different system environments.
The unified collector of the Agent of the 7xOPS system solves the problem, and executes a corresponding service line according to the obtained current system environment (including the temporarily given system environment), thereby reducing the expenditure on the development cost.
As shown in fig. 6 and 7, the performance acquisition monitoring process includes:
s1: the 7xOPS Agent task list starts a performance monitoring task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: performing Context assignment through the verified data to enable the Context assignment to be used at any time in the following monitoring task, installing a plug-in shell script and customizing a collection index by a user;
s3: running a performance monitoring task, and reading data by using Linux (acquiring built-in files) and Windows (using commands and dll files) methods to acquire data such as system equipment information cpu, a memory, a wireless network card and the like;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;
services such as intelligent deployment, system installation/upgrading/unloading and the like support automatic operation, so that human resources are reduced, the operation experience of customers is facilitated, and the difference between an internal network and an external network is realized in the services;
the 7xOPS system Agent unified collector solves the problem and supports the operations of self-starting/installing/upgrading and the like of the system in different environments.
SNMP acquisition process:
s1: the 7xOPS Agent task list starts an SNMP monitoring task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: carrying out Context assignment through the checked data so that the Context assignment can be used at any time in the following monitoring task;
s3: the SNMP monitoring task is operated, basic information such as a cpu, an internal memory and the like of the equipment can be obtained by using an SNMP walk/get request method through Oid and IP as access parameters, and data monitoring can also be carried out through a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
As shown in fig. 8, the asset scan flow:
s1: the 7xOPS Agent task list starts an asset scanning task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: through the checked data, the task Id needs to be added into a global variable Map array created by the asset scanning task, so that the received task can be started/stopped subsequently;
s3: running an asset scanning task, performing different data monitoring according to data types (icmp/SNMP) in task information, and acquiring system environment basic information by using an SNMP walk/get request method through Oid and IP as access parameters or performing data monitoring through a Ping method;
s4: carrying out data cleaning and data filtering on the monitored data, and carrying out data transmission after data integration;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Server and the 7xOPS Server to realize the storage of the acquired data;
data interaction, wherein the influence of the data volume in the internal network transmission on the data transmission bandwidth is not great, and the data loss and link timeout are caused by overlarge transmission memory in the interaction with the external network;
the 7xOPS system Agent unified collector provides a scheme of data structure optimization, transmission data compression and the like to solve the problem of data loss caused by bandwidth generated by overlarge data in external network transmission.
As shown in fig. 9 and 10, the state monitoring process:
s1: the 7xOPS Agent task list starts a state monitoring task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: adding the task Id into a global variable Map array created by the state monitoring task to enable the received task to be started and stopped subsequently;
s3: running a state monitoring task, monitoring data according to the IP in the task information, and acquiring data such as whether normal communication can be realized by using a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;
a sensor monitoring process:
s1: the 7xOPS Agent task list starts a sensor monitoring task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: adding the task Id into a global variable Map array created by the sensor monitoring task to enable the received task to be started and stopped subsequently;
s3: running a sensor monitoring task, sending equipment information to a specified queue by an IOT (input/output) equipment used by a user at regular time, starting monitoring the queue by a 7xOPS (application operating system) Agent after the task is started, and reading data by a method for monitoring an exclusive queue;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
As shown in fig. 11, the domain name detection process:
s1: the 7xOPS Agent task list starts a domain name monitoring task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: adding the task Id into a global variable Map array created by the domain name monitoring task to enable the received task to be started and stopped subsequently;
s3: running a domain name monitoring task, monitoring data according to IP in task information, and acquiring data such as packet loss, time delay, state codes and the like by using a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
As shown in fig. 12, the UCPE acquisition process:
s1: the 7xOPS Agent task list starts a UCPE monitoring task, the 7xOPS Agent receives task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: the task Id needs to be added into a global variable Map array created by the UCPE monitoring task, so that the received task can be started and stopped subsequently;
s3: running the UCPE monitoring task, executing the shell script to obtain 7xNetWork equipment information, and reading the shell script to monitor data;
s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;
data acquisition, data real-time acquisition, wherein one acquisition task sometimes needs to maintain data acquisition of more than 500 machines, and the problem of data acquisition real-time performance is solved;
the unified collector of the Agent of the 7xOPS system acquires data in a go co-process mode, the utilization rate of a server cpu is increased due to the fact that the co-process opening number is too large, optimization is carried out in a co-process pool mode, and the problem that the real-time performance of the data cannot reach the standard due to the fact that the acquired data are too large is solved.
As shown in fig. 13, the 7xOPS Agent task starts and stops: the 7xOPS Server realizes the management of the 7xOPS Agent monitoring task, and can start and stop different tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring;
when the 7xOPS Server issues the task to the 7xOPS Agent through Mq, the 7xOPS Agent runs a starting program and a stopping program according to the data analyzed in the issuing.
The 7xOPS Agent receives the task and respectively realizes monitoring services according to different task types when the task is started; when the analyzed data needs Context (Context) transmission, assigning the received message to the Context, and only assigning the task ID to a Map array generated by the corresponding monitoring task when the received message is not needed;
when the 7xOPS Agent receives that the task is stopped, stopping the monitoring service according to different task types; according to the assignment operation during starting, operating the Context/Map to interrupt the monitoring task and stop monitoring;
when the 7xOPS Agent monitoring task is completed, the timing task state maintenance can send the executed task state to the corresponding Mq queue, the 7xOPS Server is used for timing consumption, reading the information and updating the task state in the 7xOPS Agent monitoring task list in time.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (10)
1. The integrated data acquisition method for multi-source data fusion is characterized in that: the 7xOPS Agent is automatically and manually registered;
s1: realizing global initialization of a configuration file, global initialization of a print log, configuration of an agent IP pool and system initialization;
s2: requesting the 7XOPS Server to acquire Token, automatically requesting the 7XOPS Server to acquire Token every 5 minutes, and starting in a coroutine mode;
the Token generation rule is randomly generated according to a user name and a password, an assignment operation is carried out after a returned Token is obtained, and a 7xOPS Agent is assigned to an object of global configuration;
s3: performing registration verification on the acquired Token in a 7xOPS Server, and judging whether the 7xOPS Agent is automatically/manually deployed;
s31: automatically deploying a request 7xOPS Server, updating the management state of the 7xOPS Agent, returning a request result, if the request result is successful, updating the global configuration, and if the request result is failed, exiting the current function and re-initiating the request;
s32: the manual deployment requests the 7xOPS Server, newly deployed 7xOPS agents are registered, if the registration is successful, the global configuration is updated, if the registration is failed, the program is quitted, and the process is ended;
s4: the 7 xpps Agent registration is complete.
2. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: the 7xOPS Agent is automatically deployed:
s1: creating a new Agent in the 7xOPS Agent management list, wherein the new Agent needs to install an address IP, a root account number and a password, an installation path, a download link and other information;
s2: the 7xOPS Server starts SSH to establish a link, uploads a shell script after the link is established successfully, executes the shell script, jumps to a specified path, downloads an installation package file, and starts after the installation package file is downloaded successfully;
s3: the 7xOPS Agent is successfully started, the registration is completed in the 7xOPS Server, the 7xOPS Agent can regularly send the Agent state for updating, and an installation log is attached to the Agent state for sending to the Mq;
s4: and the 7xOPS Server receives the state updating message from the Mq, modifies the state of the 7xOPS Agent created in the 7xOPS Agent management list into the running state, and saves the installation log.
3. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: the 7xOPS Agent is automatically upgraded and uninstalled:
s1: clicking and upgrading in the 7xOPS Agent management list, issuing a task to a corresponding Mq queue by the 7xOPS Server, and monitoring the Mq task issued by the 7xOPS Server by the 7xOPS Agent;
s2: the 7xOPS Agent monitors the information to be upgraded, analyzes the received information, executes the shell script, downloads the installation package through the link, and judges whether the 7xOPS Agent is started or not through the checking process after the installation is finished;
s21: in the starting process: backing up a currently running installation package, killing a current program process, running a new installation package, and sending a state message to an Mq task after the successful starting;
s22: and (3) no start-up: backing up an old installation package, operating a new installation package, and sending a state message to an Mq task after the new installation package is successfully started;
s3: and the 7xOPS Server receives the status message from the Mq, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list into the running status, and saves the upgrade log.
4. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: the monitoring task acquisition:
the 7xOPS Server realizes the management of a 7xOPS Agent monitoring task, performs the starting and stopping of various tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring, and sends a monitored information template to a corresponding Mq queue according to the task type after the task is started;
the 7xOPS Agent monitors an Mq task issued by the 7xOPS Server, performs different data monitoring according to the type of the task, performs data cleaning, data filtering, data integration, data queuing, data structure optimization and data compression on the monitored data, and finally performs data transmission;
the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
5. The integrated data acquisition method for multi-source data fusion according to claim 4, wherein: the performance acquisition monitoring process comprises the following steps:
s1: the 7xOPS Agent task list starts a performance monitoring task, the 7xOPS Agent receives Mq task information issued by the 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;
s2: performing Context assignment on the verified data to enable the data to be used in the whole process in the following monitoring task, installing a plug-in shell script and customizing a collection index by a user;
s3: running a performance monitoring task, and reading data by using a Linux (acquiring built-in files) and a Windows (using commands and dll files) method to acquire data such as system equipment information cpu, an internal memory, a wireless network card and the like;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;
the SNMP acquisition process comprises the following steps:
s1: the 7xOPS Agent task list starts an SNMP monitoring task, the 7xOPS Agent receives Mq task information issued by the 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;
s2: context assignment is carried out on the checked data, so that the data can be used in the whole process in the following monitoring task;
s3: the SNMP monitoring task is operated, basic information such as a cpu, an internal memory and the like of the equipment can be obtained by using an SNMP walk/get request method through Oid and IP as access parameters, and data monitoring can also be carried out through a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
6. The integrated data acquisition method for multi-source data fusion according to claim 4, characterized in that: the asset scanning process comprises the following steps:
s1: the 7xOPS Agent task list starts an asset scanning task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;
s2: through the checked data, the task Id needs to be added into a global variable Map array created by the asset scanning task, so that the received task can be subsequently started/stopped;
s3: running an asset scanning task, performing different data monitoring according to data types (icmp/SNMP) in task information, and acquiring system environment basic information by using an SNMP walk/get request method through Oid and IP as access parameters or performing data monitoring through a Ping method;
s4: carrying out data cleaning and data filtering on the monitored data, and carrying out data transmission after data integration;
s5: and the data monitored by the 7xOPS Agent is sent to the 7xOPS Server and the 7xOPS Server to realize the storage of the acquired data.
7. The integrated data acquisition method for multi-source data fusion according to claim 4, characterized in that: the state monitoring process comprises the following steps:
s1: the 7xOPS Agent task list starts a state monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to start;
s2: adding the task Id into a global variable Map array created by the state monitoring task to enable the received task to be started and stopped subsequently;
s3: running state monitoring tasks, monitoring data according to IP in task information, and acquiring data such as whether normal communication can be realized by using a Ping method;
s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize the storage of the collected data;
the sensor monitoring process comprises the following steps:
s1: the 7xOPS Agent task list starts a sensor monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to start;
s2: adding the task Id into a global variable Map array created by the sensor monitoring task to enable the received task to be started and stopped subsequently;
s3: running a sensor monitoring task, sending equipment information to a specified queue by an IOT (input/output) equipment used by a user at regular time, starting monitoring the queue by a 7xOPS (application operating system) Agent after the task is started, and reading data by a method for monitoring an exclusive queue;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
8. The integrated data acquisition method for multi-source data fusion according to claim 4, characterized in that: the domain name detection process comprises the following steps:
s1: the 7xOPS Agent task list starts a domain name monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;
s2: adding the task Id into a global variable Map array created by the domain name monitoring task to enable the received task to be started and stopped subsequently;
s3: running a domain name monitoring task, monitoring data according to IP in task information, and acquiring data such as packet loss, time delay, state codes and the like by using a Ping method;
s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;
s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
9. The integrated data acquisition method for multi-source data fusion according to claim 4, wherein: the UCPE acquisition process comprises the following steps:
s1: the 7xOPS Agent task list starts a UCPE monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;
s2: adding the task Id into a global variable Map array created by the UCPE monitoring task to enable the received task to be started and stopped subsequently;
s3: running the UCPE monitoring task, executing the shell script to obtain 7xNetWork equipment information, and reading the shell script to monitor data;
s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;
s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.
10. The integrated data acquisition method for multi-source data fusion according to claim 4, wherein: the 7xOPS Agent task is started and stopped: the 7xOPS Server realizes the management of the 7xOPS Agent monitoring task, and can start and stop different tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring;
when the 7xOPS Server issues the task to the 7xOPS Agent through Mq, the 7xOPS Agent runs a start and stop program according to the data analyzed in the issuing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211530609.0A CN115811457A (en) | 2022-12-01 | 2022-12-01 | Integrated data acquisition method for multi-source data fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211530609.0A CN115811457A (en) | 2022-12-01 | 2022-12-01 | Integrated data acquisition method for multi-source data fusion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115811457A true CN115811457A (en) | 2023-03-17 |
Family
ID=85484633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211530609.0A Pending CN115811457A (en) | 2022-12-01 | 2022-12-01 | Integrated data acquisition method for multi-source data fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115811457A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117290671A (en) * | 2023-11-27 | 2023-12-26 | 中国西安卫星测控中心 | Data preprocessing method and device, electronic equipment and computer storage medium |
-
2022
- 2022-12-01 CN CN202211530609.0A patent/CN115811457A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117290671A (en) * | 2023-11-27 | 2023-12-26 | 中国西安卫星测控中心 | Data preprocessing method and device, electronic equipment and computer storage medium |
| CN117290671B (en) * | 2023-11-27 | 2024-02-09 | 中国西安卫星测控中心 | Data preprocessing method and device, electronic equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7181519B2 (en) | Distributed network monitoring and control system | |
| EP1168711B1 (en) | Process for controlling devices of an intranet network through the web | |
| US20220078092A1 (en) | Provisioning a service | |
| US10698923B2 (en) | Methods and apparatus for providing adaptive private network database schema migration and management processes | |
| US7219140B2 (en) | Configuration and management systems for mobile and embedded devices | |
| EP3299953B1 (en) | System and method for providing automatic firmware update management | |
| CN101778136B (en) | Method for installing operating systems for client through remote real-time monitoring of network | |
| US20030069956A1 (en) | Object oriented SNMP agent | |
| CN111786893B (en) | method and device for synchronizing software versions of wireless routers in mesh networking | |
| CN103957117A (en) | System and method for achieving intelligent installation of network equipment | |
| CN102932455B (en) | Construction method based on cloud computing render farms | |
| CN101727343A (en) | Software installation and deployment method for distributed server | |
| CN108874437A (en) | A kind of online cloud update method of android application program | |
| CN109522028A (en) | A kind of installation method and system of operating system | |
| CN115811457A (en) | Integrated data acquisition method for multi-source data fusion | |
| CN103905226B (en) | The automatic deployment method and device of a kind of business platform | |
| CN106210114A (en) | A kind of multiple terminals software parallel downloads upgrade method | |
| CN112948008B (en) | Method for managing physical bare machine based on Ironic | |
| CN114500530A (en) | Automatic adjustment method for civil edge information system | |
| WO2002039257A2 (en) | Automated provisioning framework for internet site servers | |
| CN115550764A (en) | Network element configuration method, system, equipment and medium based on automatic opening system | |
| CN113138787A (en) | Program updating method and device | |
| CN110650071B (en) | Access management method and device of user equipment and management server | |
| KR100452629B1 (en) | Apparatus and Method of Network Management by Upgrading Simple Network Management Protocol Management information Base Bundle Dynamically | |
| CN108897679A (en) | A kind of method and system of deployment services device operating system testing environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |