CN115811457A - Integrated Data Acquisition Method for Multi-source Data Fusion - Google Patents

Abstract

The invention belongs to the technical field of multi-source data fusion, and discloses an integrated data collection method for multi-source data fusion. The automatic and manual registration of the 7xOPS Agent: S1: realize the global initialization of the configuration file, the global initialization of the print log, and the configuration of the agent IP pool , system initialization; S2: request 7xOPS Server to allocate Token, automatically request 7xOPS Server to obtain Token every 5 minutes, and start in a collaborative process mode; the present invention provides an integrated data collection method for multi-source data fusion, through 7xOPS Server to 7xOPS Agent For management, operations such as installation, upgrade, uninstallation, and editing can be realized. Task types include performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring. 7xOPS Server implements operations such as start, stop, edit, and delete of tasks, implements data collection of different task types through 7xOPS Agent, and implements storage of collected data through 7xOPS Receiver.

Description

Integrated Data Acquisition Method for Multi-source Data Fusion

technical field

The invention belongs to the technical field of multi-source data fusion, in particular to an integrated data acquisition method for multi-source data fusion.

Background technique

With the popularization of broadband network and the increasing demand of management departments for management level, urban public security, transportation, finance, environmental protection, electric power, medical care, education and other management departments have the same demand for large-scale networked security monitoring platforms within the city. It started to increase a lot in two years. With the maturity of network transmission technology and the Internet, major breakthroughs have been made to meet the needs of customers in these emerging industries. Large-scale automated operation and maintenance monitoring systems are being rapidly built all over the country. The existing common collection The mainstream operating systems supported by the server include Windows, various Linux, Unix, etc., and research and development are carried out for specific operating systems. Each operating system has its own independent development environment, which has disadvantages such as high research and development costs and difficult code maintenance.

Contents of the invention

In order to solve the problems raised in the above-mentioned background technology, the present invention provides an integrated data acquisition method for multi-source data fusion. The 7xOPS Agent is managed through the 7xOPS Server, and operations such as installation, upgrade, and uninstallation can be realized. The task types include performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, UCPE monitoring and other tasks. 7xOPS Server implements operations such as deletion, editing, starting, and stopping of tasks, 7xOPS Agent implements unified entry management and data collection for different task types, and 7xOPS Receiver implements storage of collected data.

In order to achieve the above object, the present invention provides the following technical solutions: an integrated data collection method for multi-source data fusion, and the 7xOPS Agent is automatically and manually registered;

S1: Realize the global initialization of the configuration file, the global initialization of the printing log, the configuration of the proxy IP pool, and the system initialization;

S2: Request 7xOPS Server to obtain Token, automatically request 7xOPS Server to obtain Token every 5 minutes, and start in the form of coroutine;

The Token generation rule is randomly generated according to the user name and password, and the assignment operation is performed after the returned Token is obtained, and the 7xOPS Agent is assigned to the object of the global configuration;

S3: Register and verify the obtained Token with 7xOPS Server, and determine whether the 7xOPS Agent is deployed automatically or manually;

S31: Automatically deploy the request to 7xOPS Server, update the management status of 7xOPS Agent, return the request result, update the global configuration if successful, exit the current function and re-initiate the request if it fails;

S32: Manual deployment requests 7xOPS Server, registers the newly deployed 7xOPS Agent, updates the global configuration if successful, exits the program and ends the process if it fails;

S4: 7xOPS Agent registration is complete.

Preferably, the automatic deployment of the 7xOPS Agent:

S1: Create a new Agent in the 7xOPS Agent management list, which requires installation address IP, root account and password, installation path, download link and other information;

S2: 7xOPS Server starts SSH to establish a link. After the link is successfully established, upload the shell script, execute the shell script, jump to the specified path, download the installation package file, and start it after the download is successful;

S3: The 7xOPS Agent starts successfully, and the registration on the 7xOPS Server is completed. The 7xOPS Agent will regularly send the Agent status to update, and send it to Mq with the installation log;

S4: 7xOPS Server receives a status update message from Mq, modifies the status of the 7xOPS Agent created in the 7xOPS Agent management list to running, and saves the installation log.

Preferably, the 7xOPS Agent automatically upgrades and uninstalls:

S1: Click Upgrade in the 7xOPS Agent management list, the 7xOPS Server sends tasks to the corresponding Mq queue, and the 7xOPS Agent monitors the Mq tasks sent by the 7xOPS Server;

S2: 7xOPS Agent listens to the information to be upgraded, parses the received information, executes the shell script, downloads the installation package through the link, and checks the progress of the installation to determine whether the 7xOPS Agent is started;

S21: Starting: the currently running installation package is backed up, the current program process is killed, a new installation package is run, and a status message is sent to the Mq task after the startup is successful;

S22: not started: back up the old installation package, run the new installation package, and send a status message to the Mq task after the startup is successful;

S3: 7xOPS Server receives the status message from Mq, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list to running, and saves the upgrade log.

Preferably, the monitoring task collects:

7xOPS Server realizes the monitoring task management of 7xOPS Agent, and performs various tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring. The information template is sent to the corresponding Mq queue;

7xOPS Agent listens to the Mq tasks issued by 7xOPS Server, performs different data monitoring according to the type of tasks, and performs data cleaning, data filtering, data integration, data queue, data structure optimization, data compression and finally data transmission on the monitored data ;

The data monitored by 7xOPS Agent is sent to 7xOPS Receiver in the form of Http/Rpc, and 7xOPS Receiver realizes the storage of the collected data.

Preferably, the performance collection and monitoring process:

S1: The 7xOPS Agent task list starts the performance monitoring task. The 7xOPS Agent receives the Mq task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: Context assignment is carried out through the verified data, so that it can be used throughout the next monitoring task, install plug-in shell scripts, and user-defined collection indicators;

S3: Run performance monitoring tasks, use Linux (to obtain built-in files) and Windows (to use commands and dll files) to read data and obtain system device information cpu, memory, wireless network card and other data;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

The SNMP collection process:

S1: The 7xOPS Agent task list starts the SNMP monitoring task, and the 7xOPS Agent receives the Mq task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, the data verification is required;

S2: Context assignment is performed through the verified data, so that it can be used throughout the next monitoring task;

S3: Run the SNMP monitoring task, use the SNMP walk/get request method to obtain basic information such as the cpu and memory of the device through Oid and IP as input parameters, and also perform data monitoring through the Ping method;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

Preferably, the asset scanning process:

S1: The 7xOPS Agent task list starts the asset scanning task, and the 7xOPS Agent receives the Mq task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, the data verification is required;

S2: After passing the verified data, the task Id needs to be added to the global variable Map array created by the asset scanning task, so that the received task can be started/stopped subsequently;

S3: Run the asset scanning task, perform different data monitoring according to the data type (icmp/snmp) in the task information, use the SNMP walk/get request method to obtain the basic information of the system environment through Oid and IP as input parameters, or through Ping method for data monitoring;

S4: The monitored data is cleaned, filtered, and sent after data integration;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Server, and the 7xOPS Server realizes the storage of the collected data.

Preferably, the state monitoring process:

S1: The 7xOPS Agent task list starts the status monitoring task, and the 7xOPS Agent receives the Mq task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, the data verification is required;

S2: The task Id needs to be added to the global variable Map array created by the status monitoring task, so that the received task can be started and stopped subsequently;

S3: Running status monitoring tasks, data monitoring according to the IP in the task information, using the Ping method to obtain data such as whether normal communication is possible;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

The sensor monitoring process:

S1: The 7xOPS Agent task list starts the sensor monitoring task, and the 7xOPS Agent receives the Mq task information issued by the 7xOPSServer to execute the corresponding monitoring task. Before starting the monitoring task, the data verification is required;

S2: The task Id needs to be added to the global variable Map array created by the sensor monitoring task, so that the received task can be started and stopped subsequently;

S3: Run the sensor monitoring task, the IOT device used by the user sends device information to the designated queue regularly, 7xOPS Agent starts to monitor the queue after the task is started, and reads data by monitoring the exclusive queue method;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

Preferably, the domain name detection process:

S1: The 7xOPS Agent task list starts the domain name monitoring task, and the 7xOPS Agent receives the Mq task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: The task ID needs to be added to the global variable Map array created by the domain name monitoring task, so that the received task can be started and stopped subsequently;

S3: Run the domain name monitoring task, perform data monitoring according to the IP in the task information, and use the Ping method to obtain data such as packet loss, delay, and status code;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

Preferably, the UCPE acquisition process:

S1: The 7xOPS Agent task list starts the UCPE monitoring task, and the 7xOPS Agent receives the Mq task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, the data verification is required;

S2: The task ID needs to be added to the global variable Map array created by the UCPE monitoring task, so that the received task can be started and stopped subsequently;

S3: Run the UCPE monitoring task, execute the shell script to obtain 7xNetWork device information, and read the shell script to monitor the data;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

Preferably, the 7xOPS Agent task starts and stops: the 7xOPS Server realizes the monitoring task management of the 7xOPS Agent, and can start and execute different tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring stop;

When the 7xOPS Server sends the task to the 7xOPS Agent through Mq, the 7xOPS Agent starts and stops the program according to the data analyzed in the sending.

Compared with the prior art, the beneficial effects of the present invention are as follows:

The present invention provides an integrated data acquisition method for multi-source data fusion, and manages 7xOPSAgent through 7xOPS Server, which can realize operations such as installation, upgrade, and uninstallation. The task types include performance monitoring, SNMP monitoring, asset scanning, status monitoring, and sensor monitoring. , domain name monitoring, UCPE monitoring and other tasks. 7xOPS Server implements operations such as installation, upgrade, uninstallation, start, and stop of tasks, implements unified entry management and data collection for different task types through 7xOPS Agent, and 7xOPS Receiver implements storage of collected data.

Description of drawings

Fig. 1 is a schematic diagram of the architecture of the present invention;

Fig. 2 is 7xOPS Agent automatic, manual registration flow diagram of the present invention;

Fig. 3 is the schematic diagram of the automatic deployment flow chart of 7xOPS Agent of the present invention;

Fig. 4 is the 7xOPS Agent automatic upgrade of the present invention, the schematic diagram of unloading process;

Fig. 5 is a schematic diagram of the task collection process of the present invention;

Fig. 6 is a schematic diagram of the performance collection process of the present invention;

Fig. 7 is the SNMP collection flow schematic diagram of the present invention;

Fig. 8 is a schematic diagram of the asset scanning process of the present invention;

Fig. 9 is a schematic diagram of a state detection flow chart of the present invention;

Fig. 10 is a schematic diagram of the sensor detection process of the present invention;

FIG. 11 is a schematic diagram of a domain name detection process of the present invention;

Fig. 12 is a schematic diagram of the UCPE collection process of the present invention;

Fig. 13 is a schematic diagram of the process of starting and stopping the 7xOPS Agent task of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

As shown in Figures 1 to 13, the present invention provides an integrated data collection method for multi-source data fusion, an integrated data collection method for multi-source data fusion, 7xOPS Agent automatic and manual registration:

S1: Realize the global initialization of the configuration file, the global initialization of the printing log, the configuration of the proxy IP pool, and the system initialization;

S2: Request 7xOPS Server to obtain Token, automatically request 7xOPS Server to obtain Token every 5 minutes, and start in the form of coroutine;

Token generation rules, randomly generated according to the user name and password, after obtaining the returned Token, carry out the assignment operation, and assign the value to the object of the global configuration;

S3: Register and verify the obtained Token with 7xOPS Server, and determine whether the 7xOPS Agent is deployed automatically or manually;

S31: Automatically deploy the request to 7xOPS Server, update the management status of 7xOPS Agent, and return the request result, if successful, update the global configuration, and if it fails, exit the current function and make a new request;

S32: Manual deployment requests 7xOPS Server, registers the newly deployed 7xOPS Agent, updates the global configuration if successful, exits the program and ends the process if it fails;

S4: 7xOPS Agent registration is completed;

As shown in Figure 3, 7xOPS Agent is automatically deployed:

S1: Create a new monitoring task in the 7xOPS Agent management list, which requires installation address IP, root account and password, installation path, download link and other information;

S2: 7xOPS Server starts SSH to establish a link. After the link is successfully established, upload the shell script, execute the shell script, jump to the specified path, download the installation package file, and start it after the download is successful;

S3: 7xOPS Agent starts and succeeds, registration is completed in 7xOPS Server, 7xOPS Agent will regularly send monitoring task status updates, and attach installation logs;

S4: The 7xOPS Server receives the status update message, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list to running, and saves the installation log.

As shown in Figure 4, 7xOPS Agent automatically upgrades and uninstalls:

S1: Click Start in the 7xOPS Agent management list, the 7xOPS Server sends tasks to the corresponding Mq queue, and the 7xOPS Agent monitors the Mq tasks sent by the 7xOPS Server;

S2: 7xOPS Agent monitors the information to be upgraded, executes the shell script, downloads the installation package through the link, and determines whether the 7xOPS Agent is started;

S21: Starting: the currently running installation package is backed up, the current program process is killed, a new installation package is run, and a status message is sent to the Mq task after the startup is successful;

S22: not started: back up the old installation package, run the new installation package, and send a status message to the Mq task after the startup is successful;

S3: The 7xOPS Server receives the status update message, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list to running, and saves the upgrade/uninstallation log.

As shown in Figure 5, the monitoring task collection:

7xOPS Server realizes the monitoring task management of 7xOPS Agent, and performs various tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring. The information template is sent to the corresponding Mq queue;

7xOPS Agent listens to the tasks delivered by 7xOPS Server, performs different data monitoring according to the type of tasks, and performs data cleaning, data filtering, data integration, data queuing, data structure optimization, data compression and finally data transmission on the monitored data;

The data monitored by 7xOPS Agent is sent to 7xOPS Receiver in the form of Http/Rpc, and 7xOPSReceiver realizes the storage of the collected data;

7xOPS Agent is unified, suitable for monitoring most business scenarios in the market, and can also create custom templates, for example: some systems only have snmp monitoring/ping monitoring, most of them only have snmp monitoring;

"7xOPS System Agent Unified Collector" integrates tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, UCPE monitoring and custom templates in the project. It is easy to configure, convenient for users to use and unified management, etc. operate.

System compatibility, adapting to the development of Linux, Windows, Mac and other system environments. For example, due to the inconsistency of system commands in Linux, Windows, and Mac, project development needs to be independently developed in different system environments.

"7xOPS System Agent Unified Collector" solves this problem, and executes the corresponding business line according to the obtained current system environment (including the temporary system environment), which reduces the expenditure on research and development costs.

As shown in Figure 6 and Figure 7, the performance collection and monitoring process:

S1: The 7xOPS Agent task list starts the performance monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: Context assignment is carried out through the verified data, so that it can be used at any time in the next monitoring task, install the plug-in shell script, and user-defined collection indicators;

S3: Run performance monitoring tasks, use Linux (to obtain built-in files) and Windows (to use commands and dll files) to read data and obtain system device information cpu, memory, wireless network card and other data;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

Intelligent deployment, system installation/upgrade/uninstallation and other services support automated operations, reducing human resources and facilitating customers' operating experience. In the realization of these services, the difference between intranet and extranet is aimed at;

"7xOPS System Agent Unified Collector" solves this problem, and supports operations such as system self-start/installation/upgrade in different environments.

SNMP collection process:

S1: The 7xOPS Agent task list starts the SNMP monitoring task, and the 7xOPS Agent receives the task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, the data verification is required;

S2: Context assignment is performed through the verified data, so that it can be used at any time in the next monitoring task;

S3: Run the SNMP monitoring task, use the SNMP walk/get request method to obtain basic information such as the cpu and memory of the device through Oid and IP as input parameters, and also perform data monitoring through the Ping method;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

As shown in Figure 8, the asset scanning process:

S1: The 7xOPS Agent task list starts the asset scanning task, and the 7xOPS Agent receives the task information sent by the 7xOPS Server to perform the corresponding monitoring task, and then needs to pass the data verification before starting the monitoring task;

S2: After passing the verified data, the task Id needs to be added to the global variable Map array created by the asset scanning task, so that the received task can be started/stopped subsequently;

S3: Run the asset scanning task, perform different data monitoring according to the data type (icmp/snmp) in the task information, use the SNMP walk/get request method to obtain the basic information of the system environment through Oid and IP as input parameters, or through Ping method for data monitoring;

S4: The monitored data is cleaned, filtered, and sent after data integration;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Server, and the 7xOPS Server realizes the storage of the collected data;

For data interaction, the amount of data in internal network transmission does not have a great influence on the data transmission bandwidth, but in the interaction with the external network, data loss and link timeout are caused by excessive transmission memory;

"7xOPS System Agent Unified Collector" provides data structure optimization, transmission data compression and other solutions to solve the problem of data loss caused by bandwidth generated by excessive data in external network transmission.

As shown in Figure 9 and Figure 10, the status monitoring process:

S1: The 7xOPS Agent task list starts the status monitoring task, and the 7xOPS Agent receives the task information issued by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: The task Id needs to be added to the global variable Map array created by the status monitoring task, so that the received task can be started and stopped subsequently;

S3: Running status monitoring tasks, data monitoring according to the IP in the task information, using the Ping method to obtain data such as whether normal communication is possible;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

Sensor monitoring process:

S1: The 7xOPS Agent task list starts the sensor monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPSServer to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: The task Id needs to be added to the global variable Map array created by the sensor monitoring task, so that the received task can be started and stopped subsequently;

S3: Run the sensor monitoring task, the IOT device used by the user sends device information to the designated queue regularly, 7xOPS Agent starts to monitor the queue after the task is started, and reads data by monitoring the exclusive queue method;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

As shown in Figure 11, the domain name detection process:

S1: The 7xOPS Agent task list starts the domain name monitoring task, and the 7xOPS Agent receives the task information sent by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: The task ID needs to be added to the global variable Map array created by the domain name monitoring task, so that the received task can be started and stopped subsequently;

S3: Run the domain name monitoring task, perform data monitoring according to the IP in the task information, and use the Ping method to obtain data such as packet loss, delay, and status code;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data.

As shown in Figure 12, the UCPE acquisition process:

S1: The 7xOPS Agent task list starts the UCPE monitoring task. The 7xOPS Agent receives the task information sent by the 7xOPS Server to execute the corresponding monitoring task. Before starting the monitoring task, it needs to pass the data verification;

S2: The task ID needs to be added to the global variable Map array created by the UCPE monitoring task, so that the received task can be started and stopped subsequently;

S3: Run the UCPE monitoring task, execute the shell script to obtain 7xNetWork device information, and read the shell script to monitor the data;

S4: Perform data cleaning, data filtering, data integration, data queuing, data structure optimization, and data compression for the monitored data to send data;

S5: The data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver in the form of Http/Rpc, and the 7xOPS Receiver realizes the storage of the collected data;

Data collection, real-time collection of data, a collection task sometimes needs to maintain the data collection of more than 500 machines, the problem of real-time data acquisition;

"7xOPS System Agent Unified Collector" uses go coroutines to acquire data. Because the number of coroutines is too large, the CPU usage of the server increases. The coroutine pool is used for optimization, which solves the problem of data collection. If the data is too large, the real-time performance of the data is not up to standard.

As shown in Figure 13, 7xOPS Agent task start and stop: 7xOPS Server implements 7xOPS Agent monitoring task management, and can start different tasks such as performance monitoring, SNMP monitoring, asset scanning, status monitoring, sensor monitoring, domain name monitoring, and UCPE monitoring and stop;

When the 7xOPS Server sends the task to the 7xOPS Agent through Mq, the 7xOPS Agent starts and stops the program according to the data analyzed in the sending.

7xOPS Agent receives tasks to start monitoring services according to different task types; when the parsed data needs context (Context) transmission, it assigns the received message to Context, and only needs to assign the task ID to the corresponding The Map array generated by the monitoring task;

7xOPS Agent stops the monitoring service according to different task types when it receives a task to stop; according to the assignment operation at startup, it operates Context/Map to interrupt the monitoring task and stop monitoring;

When the 7xOPS Agent monitoring task is completed, the scheduled task status maintenance will send the completed task status to the corresponding Mq queue, and the 7xOPS Server will consume regularly, read the information and update the task status in the 7xOPS Agent monitoring task list in time.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (10)

1. The integrated data acquisition method for multi-source data fusion is characterized in that: the 7xOPS Agent is automatically and manually registered;

s1: realizing global initialization of a configuration file, global initialization of a print log, configuration of an agent IP pool and system initialization;

s2: requesting the 7XOPS Server to acquire Token, automatically requesting the 7XOPS Server to acquire Token every 5 minutes, and starting in a coroutine mode;

the Token generation rule is randomly generated according to a user name and a password, an assignment operation is carried out after a returned Token is obtained, and a 7xOPS Agent is assigned to an object of global configuration;

s3: performing registration verification on the acquired Token in a 7xOPS Server, and judging whether the 7xOPS Agent is automatically/manually deployed;

s31: automatically deploying a request 7xOPS Server, updating the management state of the 7xOPS Agent, returning a request result, if the request result is successful, updating the global configuration, and if the request result is failed, exiting the current function and re-initiating the request;

s32: the manual deployment requests the 7xOPS Server, newly deployed 7xOPS agents are registered, if the registration is successful, the global configuration is updated, if the registration is failed, the program is quitted, and the process is ended;

s4: the 7 xpps Agent registration is complete.

2. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: the 7xOPS Agent is automatically deployed:

s1: creating a new Agent in the 7xOPS Agent management list, wherein the new Agent needs to install an address IP, a root account number and a password, an installation path, a download link and other information;

s2: the 7xOPS Server starts SSH to establish a link, uploads a shell script after the link is established successfully, executes the shell script, jumps to a specified path, downloads an installation package file, and starts after the installation package file is downloaded successfully;

s3: the 7xOPS Agent is successfully started, the registration is completed in the 7xOPS Server, the 7xOPS Agent can regularly send the Agent state for updating, and an installation log is attached to the Agent state for sending to the Mq;

s4: and the 7xOPS Server receives the state updating message from the Mq, modifies the state of the 7xOPS Agent created in the 7xOPS Agent management list into the running state, and saves the installation log.

3. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: the 7xOPS Agent is automatically upgraded and uninstalled:

s1: clicking and upgrading in the 7xOPS Agent management list, issuing a task to a corresponding Mq queue by the 7xOPS Server, and monitoring the Mq task issued by the 7xOPS Server by the 7xOPS Agent;

s2: the 7xOPS Agent monitors the information to be upgraded, analyzes the received information, executes the shell script, downloads the installation package through the link, and judges whether the 7xOPS Agent is started or not through the checking process after the installation is finished;

s21: in the starting process: backing up a currently running installation package, killing a current program process, running a new installation package, and sending a state message to an Mq task after the successful starting;

s22: and (3) no start-up: backing up an old installation package, operating a new installation package, and sending a state message to an Mq task after the new installation package is successfully started;

s3: and the 7xOPS Server receives the status message from the Mq, modifies the status of the newly created 7xOPS Agent in the 7xOPS Agent management list into the running status, and saves the upgrade log.

4. The integrated data acquisition method for multi-source data fusion according to claim 1, characterized in that: the monitoring task acquisition:

the 7xOPS Server realizes the management of a 7xOPS Agent monitoring task, performs the starting and stopping of various tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring, and sends a monitored information template to a corresponding Mq queue according to the task type after the task is started;

the 7xOPS Agent monitors an Mq task issued by the 7xOPS Server, performs different data monitoring according to the type of the task, performs data cleaning, data filtering, data integration, data queuing, data structure optimization and data compression on the monitored data, and finally performs data transmission;

the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.

5. The integrated data acquisition method for multi-source data fusion according to claim 4, wherein: the performance acquisition monitoring process comprises the following steps:

s1: the 7xOPS Agent task list starts a performance monitoring task, the 7xOPS Agent receives Mq task information issued by the 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;

s2: performing Context assignment on the verified data to enable the data to be used in the whole process in the following monitoring task, installing a plug-in shell script and customizing a collection index by a user;

s3: running a performance monitoring task, and reading data by using a Linux (acquiring built-in files) and a Windows (using commands and dll files) method to acquire data such as system equipment information cpu, an internal memory, a wireless network card and the like;

s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;

s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data;

the SNMP acquisition process comprises the following steps:

s1: the 7xOPS Agent task list starts an SNMP monitoring task, the 7xOPS Agent receives Mq task information issued by the 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;

s2: context assignment is carried out on the checked data, so that the data can be used in the whole process in the following monitoring task;

s3: the SNMP monitoring task is operated, basic information such as a cpu, an internal memory and the like of the equipment can be obtained by using an SNMP walk/get request method through Oid and IP as access parameters, and data monitoring can also be carried out through a Ping method;

s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;

s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.

6. The integrated data acquisition method for multi-source data fusion according to claim 4, characterized in that: the asset scanning process comprises the following steps:

s1: the 7xOPS Agent task list starts an asset scanning task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;

s2: through the checked data, the task Id needs to be added into a global variable Map array created by the asset scanning task, so that the received task can be subsequently started/stopped;

s3: running an asset scanning task, performing different data monitoring according to data types (icmp/SNMP) in task information, and acquiring system environment basic information by using an SNMP walk/get request method through Oid and IP as access parameters or performing data monitoring through a Ping method;

s4: carrying out data cleaning and data filtering on the monitored data, and carrying out data transmission after data integration;

s5: and the data monitored by the 7xOPS Agent is sent to the 7xOPS Server and the 7xOPS Server to realize the storage of the acquired data.

7. The integrated data acquisition method for multi-source data fusion according to claim 4, characterized in that: the state monitoring process comprises the following steps:

s1: the 7xOPS Agent task list starts a state monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to start;

s2: adding the task Id into a global variable Map array created by the state monitoring task to enable the received task to be started and stopped subsequently;

s3: running state monitoring tasks, monitoring data according to IP in task information, and acquiring data such as whether normal communication can be realized by using a Ping method;

s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;

s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize the storage of the collected data;

the sensor monitoring process comprises the following steps:

s1: the 7xOPS Agent task list starts a sensor monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to start;

s2: adding the task Id into a global variable Map array created by the sensor monitoring task to enable the received task to be started and stopped subsequently;

s3: running a sensor monitoring task, sending equipment information to a specified queue by an IOT (input/output) equipment used by a user at regular time, starting monitoring the queue by a 7xOPS (application operating system) Agent after the task is started, and reading data by a method for monitoring an exclusive queue;

s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;

s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.

8. The integrated data acquisition method for multi-source data fusion according to claim 4, characterized in that: the domain name detection process comprises the following steps:

s1: the 7xOPS Agent task list starts a domain name monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is prepared to be started;

s2: adding the task Id into a global variable Map array created by the domain name monitoring task to enable the received task to be started and stopped subsequently;

s3: running a domain name monitoring task, monitoring data according to IP in task information, and acquiring data such as packet loss, time delay, state codes and the like by using a Ping method;

s4: carrying out data cleaning, data filtering, data integration, data queue, data structure optimization on the monitored data, and carrying out data transmission after data compression;

s5: the data monitored by the 7xOPS Agent is sent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.

9. The integrated data acquisition method for multi-source data fusion according to claim 4, wherein: the UCPE acquisition process comprises the following steps:

s1: the 7xOPS Agent task list starts a UCPE monitoring task, the 7xOPS Agent receives Mq task information issued by a 7xOPS Server to execute a corresponding monitoring task, and data verification is needed before the monitoring task is ready to be started;

s2: adding the task Id into a global variable Map array created by the UCPE monitoring task to enable the received task to be started and stopped subsequently;

s3: running the UCPE monitoring task, executing the shell script to obtain 7xNetWork equipment information, and reading the shell script to monitor data;

s4: performing data cleaning, data filtering, data integration, data queue, data structure optimization and data transmission after data compression on the monitored data;

s5: and sending the data monitored by the 7xOPS Agent to the 7xOPS Receiver and the 7xOPS Receiver in an Http/Rpc mode to realize storage of the acquired data.

10. The integrated data acquisition method for multi-source data fusion according to claim 4, wherein: the 7xOPS Agent task is started and stopped: the 7xOPS Server realizes the management of the 7xOPS Agent monitoring task, and can start and stop different tasks such as performance monitoring, SNMP monitoring, asset scanning, state monitoring, sensor monitoring, domain name monitoring and UCPE monitoring;

when the 7xOPS Server issues the task to the 7xOPS Agent through Mq, the 7xOPS Agent runs a start and stop program according to the data analyzed in the issuing.

Images