CN110018993A - A kind of data analysis system, method and monitoring analysis system - Google Patents

Abstract

A data analysis system, method and monitoring and analysis system are used to provide statistics, in-depth analysis and rapid response to monitoring data of large data volume. The system includes: ETL module, Hadoop module and MPP database. The ETL module is used to collect monitoring data, perform data cleaning on the monitoring data, obtain valid first data, and send the first data to the Hadoop module. The Hadoop module is used to back up the first data, and perform data conversion, The second data is obtained by aggregation processing and sent to the MPP database, and the ETL module is also used to call the MPP database to execute: carry out early warning analysis on the second data according to the preset early warning rules, output the early warning data, and then generate the early warning data pattern, and according to the pre-warning rules. The set hidden danger alarm rule performs a hidden danger alarm analysis on the second data, outputs the hidden danger alarm data, and then generates a hidden danger alarm data pattern.

Description

A data analysis system, method and monitoring and analysis system

technical field

The present application relates to the technical field of monitoring data analysis, and in particular, to a data analysis system, method, and monitoring and analysis system.

Background technique

With the improvement of monitoring methods and the development of network technology, the monitoring data that can be collected by the monitoring system is becoming more and more abundant, which makes the monitoring system face the processing needs of massive data and the need for in-depth analysis. Most of the existing monitoring systems use real-time monitoring Mainly, it only has simple statistical analysis functions, and cannot provide complex statistical and in-depth analysis functions. Moreover, it does not have the characteristics of big data processing and analysis, and cannot quickly respond to incoming data.

In order to promote the centralization and integration of the support system, explore new technologies such as cloud computing and big data to meet the continuous development of the system, combined with the above requirements, it is necessary to propose a centralized processing and in-depth analysis of the data of the existing monitoring system. A system that works.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a data analysis system, method, and monitoring and analysis system, which are used to provide statistics, in-depth analysis, and rapid response to monitoring data with a large amount of data.

The purpose of this application is achieved through the following technical solutions:

In a first aspect, a data analysis system is provided. The data analysis system includes a data cleaning (Extract-Transform-Load, ETL) module, a distributed computing (Hadoop) module, and a Massively Parallel Processor (MPP) database. The ETL module is used to collect monitoring data in at least one monitoring system according to a preset period, and perform data cleaning on the monitoring data according to a preset cleaning rule, so as to obtain effective first data for use by the data analysis system. data, and send the first data to the Hadoop module, where the Hadoop module is used to back up the first data, perform data conversion and aggregation processing on the first data to obtain second data, The second data is sent to the MPP database, and the ETL module is also used to call the MPP database to execute: carry out early warning analysis on the second data according to a preset early warning rule, output early warning data, and The early-warning data, the pre-stored relevant first historical data, and the pre-set first data model generate an early-warning data model, and perform a hidden-risk alarm analysis on the second data according to the preset hidden-risk alarm rules, output the hidden-risk alarm data, and according to the hidden danger alarm rules. The hidden danger warning data, the pre-stored relevant second historical data, and the pre-set second data model generate a hidden danger warning data pattern.

Wherein, the first data model and the second data model are used to represent constraint rules between data.

In the embodiment of the present application, the cluster features of Hadoop are fully utilized, and the computing and storage requirements of each module in the data analysis system that require huge computing power are extended to each node in the Hadoop cluster, and the parallel computing and storage capabilities of the cluster are used to carry out Monitoring data mining processing work, using Hadoop to store, analyze and process huge amounts of data, combined with MPP database that can be processed in parallel to achieve statistics, in-depth analysis and rapid response to monitoring data of large amounts of data.

In a possible implementation manner, the Hadoop module includes a data source layer, an Operational Data Store (ODS) layer, and a data processing layer, the data source layer includes at least one data server, and the ODS layer includes In the distributed file system HDFS, the data processing layer includes a hive submodule and a Mapreduce submodule. The Hadoop module obtains the first data through the at least one data server, backs up the first data through the HDFS, stores the historical data in the MPP database, and supports the historical data in the MPP database. archiving; the Hadoop module also loads the first data in the HDFS through the hive sub-module, and performs data conversion and aggregation processing on the first data through the Mapreduce sub-module to obtain second data, and the The second data is stored in the HDFS.

Through the above method, using the characteristics of high efficiency, high fault tolerance, high expansion, high reliability and open source of Hadoop in massive data processing, and using the advantages of high fault tolerance and high scalability of HDFS, users are allowed to deploy Hadoop in ordinary and low-cost On the hardware, a distributed system is formed. Mapreduce provides the development of parallel applications, and realizes distributed computing and parallel task processing on the cluster. HDFS provides support for file operations and storage in the process of Mapreduce task processing.

In a possible implementation manner, the data analysis system further includes an interface module and an analysis and display module.

The interface module is configured to provide an interface for accessing the ETL module, the MPP database and the Hadoop module, and send the early warning data pattern and the hidden danger warning data pattern to the analysis and display module. The analysis and display module is used to perform data mining analysis according to the preset first data mining rule and the early warning data pattern, and output the early warning statistical results that are mined and related to the monitoring data according to the data mining analysis results. , and perform data mining analysis according to the preset second data mining rules and the hidden danger warning data pattern, output the hidden danger warning statistics results that are mined related to the monitoring data according to the data mining analysis results, and display to the user The early warning statistical results and the hidden danger warning statistical results.

Through the above method, the data analysis system can send the results obtained by the analysis to the analysis and display module through the interface module, so that the analysis and display module can perform further data mining processing according to the analysis results, and obtain statistical results that can be directly presented to the user (for example, , charts, web interface, etc.), enabling users to view data analysis results more intuitively and further improve user experience.

In a possible implementation manner, the data analysis system further includes a full-text search database.

The full-text search database is configured to provide a query function for the unstructured data included in the first data.

Through the above method, for some unstructured data (eg, PDF documents) obtained from monitoring data, the data analysis system cannot perform in-depth analysis on this part of the data, but can query this part of the data through the full-text search database.

It should be noted that, in this application, the retrieval of unstructured data can be realized by setting an ElasticSearch search server in the full-text search database, and of course, the retrieval of unstructured data can also be realized by setting other search servers in the full-text search database. Search, this application is not limited.

In a possible implementation manner, the data analysis system further includes a self-monitoring module.

The self-monitoring module is used to monitor the running time, current state and whether the operation is successful within the data analysis system, and is also used to monitor the hardware of the ETL module, the Hadoop module and the MPP database.

Through the above method, the data analysis system can monitor the running status of its own hardware and software in real time according to the self-monitoring module, and then can provide a solution for the abnormality in time when its own software and hardware are abnormal.

In a possible implementation manner, the data analysis system further includes a configuration module.

The configuration module is configured to provide permission configuration for different types of roles created by the data analysis system, and to provide mapping configurations between different types of users who use the data analysis system and the different types of roles.

The different types of roles are created according to the different types of users, and the different types of roles are in one-to-one correspondence with the different types of users.

Through the above method, the roles created by the data analysis system for different types of users are configured, so that the needs of different types of users can be met. For example, suppose that the data analysis system supports four types of users: ordinary business personnel (including business analysts, department managers, company managers, and group managers), senior business personnel, data analysis experts, and system administrators. The data analysis system Four types of roles will be created one by one for these four types of users. Through this configuration module, the rights configuration of these four types of roles can be realized. For example, the rights of ordinary business personnel can only be configured to view analysis and display. The content displayed by the module can be configured with the authority of senior business personnel. On the basis of viewing the content displayed by the analysis and display module, the content of the personalized report of the analysis and display module can be customized, and the authority of data analysis experts can be configured to be able to view at the same time. The content displayed by the analysis and presentation module and the original data received by the interface module can be configured with the authority of the system administrator to be able to perform configuration management.

In a possible implementation manner, the data analysis system further includes a management module.

The management module is configured to manage the different types of users and different user rights corresponding to the different types of users.

Through the above method, the system administrator can manage the different types of users and different user rights corresponding to the different types of users, and can flexibly set the rights for different types of users according to the needs of different types of users.

In a second aspect, an embodiment of the present application provides a data analysis method, the method includes: collecting monitoring data in at least one monitoring system according to a preset period through an ETL module, and analyzing the monitoring data according to a preset cleaning rule Perform data cleaning to obtain valid first data for use by the data analysis system, and send the first data to the distributed computing Hadoop module; back up the first data through the Hadoop module, and perform a Perform data conversion and aggregation processing on data to obtain second data, and send the second data to the massively parallel processing MPP database; call the MPP database through the ETL module to execute: the second data is processed according to the preset warning rules Perform early warning analysis, output early warning data, and generate early warning data patterns according to the early warning data, the pre-stored relevant first historical data, and the pre-set first data model, and analyze the second data according to the pre-set hidden danger warning rules. The hidden danger alarm analysis is performed, the hidden danger alarm data is output, and the hidden danger alarm data pattern is generated according to the hidden danger alarm data, the pre-stored relevant second historical data and the preset second data model.

Wherein, the first data model and the second data model are used to represent constraint rules between data.

In a third aspect, embodiments of the present application provide a monitoring and analysis system, including the first aspect and the data analysis system in any possible implementation manner included in the first aspect, and at least one monitoring system.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including program code, which, when the program code runs on a monitoring and analysis system, enables the monitoring and analysis system to execute the method described in the second aspect above. step.

In the above-mentioned embodiments of the present application, the cluster features of Hadoop are fully utilized to extend the computing and storage requirements of each module in the data analysis system that requires huge computing power to each node in the Hadoop cluster, and the parallel computing and storage capabilities of the cluster are used to Perform monitoring data mining and processing work, use Hadoop to store, analyze and process huge amounts of data, and combine with MPP database that can be processed in parallel to achieve statistics, in-depth analysis and rapid response to monitoring data of large amounts of data.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic structural diagram of a data analysis system according to an embodiment of the present application;

2 is a schematic structural diagram of a Hadoop module provided by an embodiment of the present application;

Figure 3 is a schematic diagram of backing up data to HDFS;

4 is a schematic structural diagram of another data analysis system provided by an embodiment of the present application;

5 is a schematic diagram of an early warning process provided in an embodiment of the present application;

FIG. 6 is a schematic flowchart of a hidden danger alarm provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a user operation flow provided by an embodiment of the present application;

8 is a schematic diagram of a user performing data model maintenance according to an embodiment of the present application;

9 is a schematic diagram of a user performing data model monitoring according to an embodiment of the present application;

10 is a schematic flowchart of processing Capes data by a Hadoop module provided by an embodiment of the present application;

11 is a schematic flowchart of a network-wide monitoring data synchronization process provided by an embodiment of the present application;

12 is a schematic flowchart of a flow monitoring data synchronization process provided by an embodiment of the present application;

FIG. 13A is a schematic diagram of a Capes data synchronization flowchart provided by an embodiment of the present application;

FIG. 13B is a schematic diagram of a Capes detailed data processing flow diagram provided by an embodiment of the present application;

FIG. 14 is a schematic diagram of a configuration item data synchronization process provided by an embodiment of the present application;

FIG. 15 is a schematic structural diagram of another data analysis system provided by an embodiment of the present application;

16 is a schematic structural diagram of another data analysis system provided by an embodiment of the present application;

FIG. 17 is a flowchart of a data analysis method provided by an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

In order to realize statistics, in-depth analysis and quick response to monitoring data of large amount of data, the embodiments of the present application provide a data analysis system, method and monitoring and analysis system.

FIG. 1 is a schematic structural diagram of a data analysis system provided by an embodiment of the present application. Referring to FIG. 1 , the system includes: an ETL module, a Hadoop module, and an MPP database.

The ETL module is used to collect monitoring data in at least one monitoring system according to a preset period, and perform data cleaning on the monitoring data according to a preset cleaning rule to obtain valid first data for use by the data analysis system, and convert the first data into Sent to the Hadoop module.

In the embodiments of the present application, a monitoring scenario in the communication field is used as an example for description below. In the communication field, the at least one monitoring system may be a network-wide monitoring system, a customer-aware monitoring system, etc., wherein the network-wide monitoring system is used to monitor the operation of the entire communication network, and the customer-aware monitoring system is used to monitor the user's network behavioral habits, etc.

In the embodiment of the present application, the above-mentioned preset period for collecting monitoring data in at least one monitoring system can be set according to actual needs, for example, it can be collected on a daily basis, or can be collected on a fixed time frequency. Of course, for Different monitoring systems can be set with the same collection period or with different collection periods, which is not limited in this application.

In the embodiment of the present application, the types and structures of data collected from different monitoring systems are different, and data cleaning can be performed on the monitoring data through the ETL module according to the preset cleaning rules, mainly performing consistency processing on the monitoring data, and finally obtaining The processed valid first data.

In the embodiment of the present application, the ETL module may adopt the datastage platform, which is not limited in the present application. When the ETL module adopts the datastage platform, use the quality stage in the datastage to clean the monitoring data to obtain valid first data.

The Hadoop module is used to back up the first data, perform data conversion and aggregation processing on the first data to obtain second data, and send the second data to the MPP database.

In the embodiment of the present application, the Hadoop module has two functions. One function is to back up the first data for subsequent use. Another function is to perform preprocessing calculation on the first data. For example, data conversion can be performed on the first data, the formats of monitoring data in different encoding formats can be unified, and the first data can be aggregated to obtain the second data. data.

The following is an example of performing aggregation processing and summarizing on the detailed data of traffic monitoring. First, the detailed data of flow monitoring is explained. The detailed data of flow monitoring mainly has the following purposes:

Business feature analysis: mainly for business volume, performance, and browser analysis.

Early warning/hidden danger alarm analysis: The analysis is mainly done by comparing the success of business processing and the volume of business processing.

Since the detailed data of traffic monitoring is at the millisecond level and the amount of data is large, Hadoop needs to perform 1-second aggregation processing on the detailed data of traffic monitoring, and aggregate it to obtain the second data for ETL use.

It should be noted that, when performing aggregation processing on data, the aggregation granularity is not limited. For example, the above-mentioned aggregation processing can be performed for 1 second of millisecond-level traffic monitoring detailed data, and of course, 2 seconds of aggregation processing can also be performed. This application The aggregate particle size is not limited.

The ETL module is also used to call the MPP database to perform early warning analysis on the second data according to the preset early warning rules, output the early warning data, and generate the early warning data according to the early warning data, the pre-stored relevant first historical data and the pre-set first data model. Early warning data mode, and carry out hidden danger alarm analysis on the second data according to the preset hidden danger alarm rules, output the hidden danger alarm data, and generate hidden dangers according to the hidden danger alarm data, the pre-stored relevant second historical data and the preset second data model. Alarm data mode.

Wherein, the first data model and the second data model are used to represent constraint rules between data.

In this embodiment of the present application, since the MPP database has a limited storage capacity, it cannot be used to store all the second data. Therefore, in the present application, early warning data related to early warning, hidden danger warning and hidden danger warning data are selected for storage. However, in the embodiment of the present application, for the convenience of the upper-layer application, the early warning data mode and the hidden danger alarm data mode are adopted in the MMP database to store the early warning data and the hidden danger alarm data.

Referring to FIG. 2, a schematic structural diagram of a Hadoop module provided by an embodiment of the present application, as shown in FIG. 2, in this structure, the Hadoop module includes a data source layer, an ODS layer, and a data processing layer, and the data source layer includes at least A data server, as shown in Figure 2, includes the ETL data server and the MPP data server as an example, the ODS layer includes HDFS, and the data processing layer includes the hive sub-module and the Mapreduce sub-module. The Hadoop module obtains the first data through the ETL data server, obtains the historical data in the MPP database through the MPP data server, backs up the first data through HDFS, stores the historical data in the MPP database, and supports the archiving of the historical data in the MPP database. The Hadoop module also loads the first data in HDFS through the hive submodule, and performs data transformation and aggregation processing on the first data through the Mapreduce submodule to obtain second data, and stores the second data in HDFS.

In this embodiment of the present application, the data server at the data source layer may further include an ES data server, and the ES data server may also back up data to HDFS. Specifically, as shown in FIG. 3 , by configuring a Hadoop Client (not a Hadoop cluster), upload files to HDFS through hadoop fs put command, ETL data server can directly write historical data to HDFS through DataStage's own BDFS component, ES data server can directly write data to HDFS through its own GateWay component, ES The backup is a full backup, only one historical version is retained, and the previous (for example, the previous month) backup data is deleted after each successful backup.

In the embodiment of the present application, before presenting the early warning data stored in the early warning data mode and the hidden danger warning data stored in the hidden danger warning data mode to the user, it is necessary to mine and analyze these data webs, and finally display the data in the form of charts or web pages. Intuitively presented to the user.

In the embodiment of the present application, the data analysis system further includes an analysis and display module and an interface module. Referring to FIG. 4 , the interface module is used to provide the analysis and display module with an interface to access the ETL module, the MPP database and the Hadoop module, and the early warning data The pattern and the hidden danger warning data pattern are sent to the analysis and display module, and the analysis and display module is used to perform data mining analysis according to the preset first data mining rules and early warning data patterns, and output the mined related data according to the data mining analysis results. Monitor the early warning statistics of the data, and perform data mining analysis according to the preset second data mining rules and hidden danger alarm data patterns, and output the mined hidden danger alarm statistics related to the monitoring data according to the data mining analysis results, and send them to the monitoring data. The user displays the warning statistics and the hidden danger alarm statistics.

It should be noted that the early warning statistical results and the hidden danger warning statistical results may be in the form of a graph or a web form.

In the embodiment of the present application, the following takes a complete early warning process as an example to describe the function of the data analysis system in the present application. Referring to FIG. 5 , FIG. 5 is a schematic diagram of the early warning process provided by the embodiment of the present application. It is assumed that the monitoring data collected by the ETL module from the monitoring system includes platform performance data, Capes traffic data, and traffic monitoring traffic data. After the ETL module collects these data, it performs data cleaning on these data according to the pre-set cleaning rules, and obtains data for supply. The valid data used by the data analysis system, and send the valid data to the Hadoop module for backup, aggregation, conversion and other processing, and send the processed data to the MPP database for storage. The ETL module calls the MPP database to perform the following operations: The set pre-warning rules perform pre-warning analysis on the data in the MPP database, output the pre-warning data, and generate the pre-warning data pattern according to the pre-warning data, the pre-stored relevant first historical data and the pre-set first data model, and pass the pre-warning data pattern through the interface module. Send it to the analysis and display module, and the analysis and display module performs data mining analysis according to the preset first data mining rules and early warning data patterns, and outputs the mined early warning statistical results related to the monitoring data according to the data mining analysis results, and Display the warning statistics to users.

It should be noted that the Capes data refers to data related to the operation management indicators of the communication service support system.

In the embodiment of the present application, the following takes a complete hidden danger alarm process as an example to describe the functions of the data analysis system in the present application. Referring to FIG. 6 , FIG. 6 is a schematic diagram of the hidden danger alarm process provided by the embodiment of the application. It is assumed that the monitoring data collected by the ETL module from the monitoring system includes platform performance data, Capes traffic data, and traffic monitoring traffic data. After the ETL module collects these data, it performs data cleaning on these data according to the pre-set cleaning rules, and obtains data for supply. The valid data used by the data analysis system, and send the valid data to the Hadoop module for backup, aggregation, conversion and other processing, and send the processed data to the MPP database for storage. The ETL module calls the MPP database to perform the following operations: The set hidden danger alarm rules perform hidden danger alarm analysis on the data in the MPP database, output the hidden danger alarm data, and generate the hidden danger alarm data pattern according to the hidden danger alarm data, the pre-stored relevant second historical data, and the pre-set second data model. The alarm data pattern is sent to the analysis and display module through the interface module. The analysis and display module conducts data mining analysis according to the preset second data mining rules and hidden danger alarm data pattern, and outputs the mined data related to the monitoring according to the data mining analysis results. The statistics of hidden danger alarms of data are displayed to users.

In the embodiment of the present application, the analysis and display module may display the statistical results to the user in the form of a web page. Referring to FIG. 7 , FIG. 7 is a schematic diagram of the user operation process provided by the embodiment of the present application. The analysis and display module displays the web interface to the user. After that, the user can select the module corresponding to the information the user wants to know in the main menu of the web interface. For example, in Figure 7, the user selects "Hazardous Alarm Analysis" under "Monitoring and Operation" in the main menu, and then displays the hidden danger alarm to the user. After entering the hidden danger alarm analysis interface, the content that can be displayed to the user in this interface includes: hidden danger alarm statistics interface and hidden danger alarm details display interface. By clicking the hidden danger alarm details display interface, you can further view the information interface of the hidden danger alarm details. , through the hidden danger alarm statistics interface, you can query the hidden danger alarm statistics, and through the hidden danger alarm details display interface, you can query the hidden danger alarm details.

In this embodiment of the present application, the analysis and presentation module can also support interaction with users, and users can upload data models through this module, and can also deploy data models through this module. Referring to FIG. 8, FIG. 8 is a schematic diagram of a user performing data model maintenance according to an embodiment of the present application. After the analysis and display module displays the web interface to the user, the user selects “Model Maintenance” under “Analysis and Modeling” in the main menu. ”, and then display the model upload interface and model query result interface to the user. Users can click the model query result interface to further view the model parameter configuration interface and model query filter interface. The above interfaces support user visualization processing. The parameter configuration interface sets the model's running start date, end date, period, fixed parameters and parameter values, model types and other parameters.

In this embodiment of the present application, the analysis and display module may also support monitoring data models. Referring to FIG. 9 , FIG. 9 is a schematic diagram of data model monitoring performed by a user according to an embodiment of the present application. After the analysis and display module displays the web interface to the user , the user selects "Model Monitoring" under "Analysis and Modeling" in the main menu, and then displays the model status monitoring interface and the model historical result query interface to the user. The user can click the corresponding interface to view the required content. For example, the user can Monitor and view the current release and running status of the data model, including model release time, status, model running start time, end time, success flag, etc., and the model can be activated or stopped.

In the embodiment of the present application, the function of the Hadoop module is described below by taking the processing of the detailed data of the Capes by the Hadoop module as an example. Referring to FIG. 10 , FIG. The sub-module writes a Map/Reduce program to calculate the duration of each link and the duration of a work order. First, input the original Capes data into the Map/Reduce program, read the original Capes data line by line, parse the content of each line of the original Capes data, and obtain the data of each field. value, calculate the duration of each link and the duration of the work order, and save the calculation results to HDFS. The MPP database can import the data on HDFS into the MPP database through EXTERNAL TABLE.

It should be noted that the ETL module collects monitoring data in at least one monitoring system according to a preset period, which is essentially to synchronize the data in the data analysis system and the monitoring system. In the embodiments of the present application, structured data is mainly processed, and the data synchronization process is described below by taking four typical structured data as examples.

The synchronization process of network-wide monitoring data is described below. Referring to FIG. 11 , FIG. 11 is a schematic diagram of a network-wide monitoring data synchronization process provided by an embodiment of the present application. The ETL module synchronizes network-wide monitoring data from the entire network in a preset first cycle. The monitoring system is synchronized to the ETL module, and the synchronized data is backed up to the Hadoop module at the same time, and the entire network monitoring data is synchronously stored in the MPP database for storage, ready for subsequent processing by the ETL module.

It should be noted that, the MPP database in this application may be a Vertica database, and certainly may be other databases that can be processed in parallel, which is not limited in this embodiment of the application.

The synchronization process of traffic monitoring data is described below. Referring to FIG. 12, FIG. 12 is a schematic diagram of a flow monitoring data synchronization process provided by an embodiment of the present application. The ETL module synchronizes the traffic monitoring data from the traffic monitoring data source in a preset second cycle. To the ETL module, the synchronized data is backed up to the Hadoop module at the same time, and the flow monitoring data is synchronously stored in the MPP database for storage for subsequent processing by the ETL module.

It should be noted that the above-mentioned preset second period can be set according to actual needs. For example, it can be fifteen minutes. If the second period is fifteen minutes, the ETL module is synchronized from the traffic monitoring data source every fifteen minutes. The traffic monitoring data of the last fifteen minutes, for example, needs to be synchronized with the traffic monitoring data of 00:00-00:15 at 00:16-00:30. Due to the complexity of the traffic monitoring data, after the ETL module obtains the traffic monitoring data, the Hadoop Client on the ETL module can use the Hadoop command to upload the traffic monitoring data to HDFS in the Hadoop module for processing and calculation, and save the calculation results in HDFS , the MPP database maps to the result file by dynamically creating a temporary virtual table, and then copies the data of the virtual table to the business table for the analysis and display module to use and analyze.

It should be further noted that, in this application, the traffic monitoring data can be stored through the RUM Oracle database, and of course, it can also be stored through other databases, which is not limited in this application.

The synchronization process of the Capes data is described below. Referring to FIG. 13A, FIG. 13A is a schematic flowchart of the Capes data synchronization process provided by the embodiment of the application. The ETL module synchronizes the Capes data from the network-wide monitoring system to the ETL module in a preset third cycle. , and back up the synchronized data to the Hadoop module at the same time, and store the Capes data synchronously in the MPP database for storage, ready for subsequent processing by the ETL module.

It should be noted that the Capes data may include Capes variable data and/or Capes detailed data. If the Capes data is Capes variable data, the preset third period can be set to fifteen minutes. If the Capes data is Capes detailed data, Then, the above-mentioned preset third period may be set as one day.

It should be noted that if the Capes data is Capes detailed data, after the ETL module synchronizes the Capes detailed data from the Capes data source to the ETL module, it needs to be sent to the Hadoop module for further processing, as shown in Figure 13B, Figure 13B is As shown in FIG. 13B, the ETL module synchronizes the Capes detailed data from the Capes data source to the ETL module, and then backs up the synchronized data to the Hadoop module at the same time. Because the Capes detailed data It is more complicated, so after the ETL module obtains the detailed data of the Capes, it can upload the detailed data of the Capes to the HDFS in the Hadoop module through the Hadoop Client on the ETL module for processing and calculation, and save the calculation results in the HDFS and MPP databases. Then dynamically create a temporary virtual table to map to the result file, and then copy the data of the virtual table to the business table for the analysis and display module to use and analyze.

The synchronization process of configuration item data is described below. Referring to FIG. 14, FIG. 14 is a schematic diagram of the configuration item data synchronization process provided by the embodiment of the application. The ETL module synchronizes the configuration item data from the network-wide monitoring system in a preset fourth cycle. To the ETL module, and back up the synchronized data to the Hadoop module at the same time, and store the configuration item data in the MPP database synchronously for storage for subsequent processing by the ETL module.

It should be noted that the above-mentioned preset fourth period can be set according to actual needs. For example, the fourth period can be set to one month. For example, if the synchronization operation is performed on this type of data on the 11th of each month, the synchronization method can be adopted. Full synchronization.

In this embodiment of the present application, the data analysis system further includes a full-text search database. Referring to FIG. 15 , the full-text search database is used to provide a query function for the unstructured data included in the first data.

It should be noted that, in the embodiments of this application, unstructured data mainly refers to PDF documents. For this kind of data, the data analysis system cannot carry out in-depth analysis, but this part of the data can be queried through the full-text search database.

It should be noted that, in this application, the retrieval of unstructured data can be realized by setting an ElasticSearch search server in the full-text search database, and of course, the retrieval of unstructured data can also be realized by setting other search servers in the full-text search database. Search, this application is not limited.

In the embodiment of the present application, the data analysis system further includes a self-monitoring module, as shown in FIG. 16 , the self-monitoring module is used to monitor the running time, current state and whether the operation is successful within the data analysis system executing tasks, and is also used to monitor the execution of tasks within the data analysis system. ETL module, Hadoop module and MPP database for hardware monitoring.

Through the above method, the data analysis system can monitor the running status of its own hardware and software in real time according to the self-monitoring module, and then can provide a solution for the abnormality in time when its own software and hardware are abnormal.

In the embodiment of the present application, the data analysis system further includes a configuration module, which is used to provide permission configuration for different types of roles created by the data analysis system, and to provide mapping configuration between different types of users using the data analysis system and different types of roles.

Among them, different types of roles are created according to different types of users, and different types of roles are in one-to-one correspondence with different types of users.

Through the above method, the roles created by the data analysis system for different types of users are configured, so that the needs of different types of users can be met. For example, suppose that the data analysis system supports four types of users: ordinary business personnel (including business analysts, department managers, company managers, and group managers), senior business personnel, data analysis experts, and system administrators. The data analysis system Four types of roles will be created one by one for these four types of users. Through this configuration module, the rights configuration of these four types of roles can be realized. For example, the rights of ordinary business personnel can only be configured to view analysis and display. The content displayed by the module can be configured with the authority of senior business personnel. On the basis of viewing the content displayed by the analysis and display module, the content of the personalized report of the analysis and display module can be customized, and the authority of data analysis experts can be configured to be able to view at the same time. The content displayed by the analysis and presentation module and the original data received by the interface module can be configured with the authority of the system administrator to be able to perform configuration management.

In the embodiment of the present application, the data analysis system further includes a management module for managing different types of users and different user rights corresponding to different types of users.

Through the above method, the system administrator can manage the different types of users and different user rights corresponding to the different types of users, and can flexibly set the rights for different types of users according to the needs of different types of users.

Based on the same inventive concept as the data analysis system provided above, an embodiment of the present application also provides a data analysis method based on the data analysis system, which will be described in detail below.

Referring to FIG. 17 , a flowchart of a data analysis method provided in an embodiment of the present application, as shown in FIG. 17 , includes:

S101: Collect monitoring data in at least one monitoring system according to a preset period through an ETL module, and perform data cleaning on the monitoring data according to a preset cleaning rule to obtain valid first data for use by a data analysis system, and perform data cleaning on the monitoring data according to a preset cleaning rule. A data is sent to the distributed computing Hadoop module.

S102: Back up the first data by using the Hadoop module, perform data conversion and aggregation processing on the first data to obtain second data, and send the second data to the massively parallel processing MPP database.

S103: Call the MPP database through the ETL module to execute: carry out early warning analysis on the second data according to the preset early warning rules, output the early warning data, and generate the warning data according to the early warning data, the pre-stored relevant first historical data and the pre-set first data model Early warning data mode, and carry out hidden danger alarm analysis on the second data according to the preset hidden danger alarm rules, output the hidden danger alarm data, and generate hidden dangers according to the hidden danger alarm data, the pre-stored relevant second historical data and the preset second data model. Alarm data mode.

Wherein, the first data model and the second data model are used to represent constraint rules between data.

In the above-mentioned embodiments of the present application, the cluster features of Hadoop are fully utilized to extend the computing and storage requirements of each module in the data analysis system that requires huge computing power to each node in the Hadoop cluster, and the parallel computing and storage capabilities of the cluster are used to Perform monitoring data mining and processing work, use Hadoop to store, analyze and process huge amounts of data, and combine with MPP database that can be processed in parallel to achieve statistics, in-depth analysis and rapid response to monitoring data of large amounts of data.

An embodiment of the present application provides a monitoring and analysis system, including the above-mentioned data analysis system and at least one monitoring system.

It should be noted that the data analysis system in the embodiments of the present application may be integrated into an existing monitoring system, and of course, a new monitoring and analysis system may be formed with the existing monitoring system, which is not limited in this application.

An embodiment of the present application provides a computer-readable storage medium, including program code, which, when the program code runs on an electronic device, causes the electronic device to execute the steps of any one of the above method embodiments.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

While the preferred embodiments of the present application have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of this application.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (10)

1. a data analysis system, is characterized in that, comprises: ETL module, distributed computing Hadoop module and large-scale parallel processing MPP database; The ETL module is used to collect monitoring data in at least one monitoring system according to a preset period, and perform data cleaning on the monitoring data according to a preset cleaning rule, so as to obtain effective first data for use by the data analysis system. data, and send the first data to the Hadoop module; the Hadoop module, configured to back up the first data, perform data conversion and aggregation processing on the first data to obtain second data, and send the second data to the MPP database; The ETL module is also used to call the MPP database to execute: Carry out early warning analysis on the second data according to the preset early warning rules, output the early warning data, and generate the early warning data pattern according to the early warning data, the pre-stored relevant first historical data and the pre-set first data model, and according to The preset hidden danger alarm rule analyzes the hidden danger alarm on the second data, outputs the hidden danger alarm data, and generates the hidden danger alarm data according to the hidden danger alarm data, the pre-stored relevant second historical data and the preset second data model. model; Wherein, the first data model and the second data model are used to represent constraint rules between data. 2. The system according to claim 1, wherein the Hadoop module comprises a data source layer, an operational data storage ODS layer and a data processing layer, the data source layer comprises at least one data server, and the ODS layer comprises Distributed file system HDFS, the data processing layer includes a hive submodule and a Mapreduce submodule; The Hadoop module obtains the first data through the at least one data server, backs up the first data through the HDFS, stores historical data in the MPP database, and supports archiving of historical data in the MPP database; The Hadoop module also loads the first data in the HDFS through the hive sub-module, and performs data conversion and aggregation processing on the first data through the Mapreduce sub-module to obtain second data, and the second data is obtained. Data is stored to the HDFS. 3. The system according to claim 1 or 2, further comprising an interface module and an analysis and display module; The interface module is configured to provide an interface for accessing the ETL module, the MPP database and the Hadoop module, and send the early warning data pattern and the hidden danger warning data pattern to the analysis and display module; The analysis and display module is used to perform data mining analysis according to the preset first data mining rule and the early warning data pattern, and output the early warning statistical results that are mined and related to the monitoring data according to the data mining analysis results. , and perform data mining analysis according to the preset second data mining rules and the hidden danger warning data pattern, output the hidden danger warning statistics results that are mined related to the monitoring data according to the data mining analysis results, and display to the user The early warning statistical results and the hidden danger warning statistical results. 4. The system of claim 3, further comprising a full-text search database; The full-text search database is configured to provide a query function for the unstructured data included in the first data. 5. The system of claim 4, further comprising a self-monitoring module; The self-monitoring module is used to monitor the running time, current state and whether the operation is successful within the data analysis system, and is also used to monitor the hardware of the ETL module, the Hadoop module and the MPP database. 6. The system of claim 5, further comprising a configuration module; The configuration module is configured to provide permission configurations for different types of roles created by the data analysis system, and to provide mapping configurations between different types of users who use the data analysis system and the different types of roles, wherein the different types of roles Type roles are created according to the different types of users, and the different types of roles are in one-to-one correspondence with the different types of users. 7. The system of claim 6, further comprising a management module; The management module is configured to manage the different types of users and different user rights corresponding to the different types of users. 8. A data analysis method based on the data analysis system according to any one of claims 1 to 7, characterized in that, comprising: Through the ETL module, monitoring data in at least one monitoring system is collected according to a preset period, and data cleaning is performed on the monitoring data according to a preset cleaning rule to obtain valid first data for use by the data analysis system, and sending the first data to the distributed computing Hadoop module; Back up the first data by using the Hadoop module, perform data conversion and aggregation processing on the first data to obtain second data, and send the second data to the massively parallel processing MPP database; Call the MPP database through the ETL module to execute: Carry out early warning analysis on the second data according to the preset early warning rules, output the early warning data, and generate the early warning data pattern according to the early warning data, the pre-stored relevant first historical data and the pre-set first data model, and according to The preset hidden danger alarm rule analyzes the hidden danger alarm on the second data, outputs the hidden danger alarm data, and generates the hidden danger alarm data according to the hidden danger alarm data, the pre-stored relevant second historical data and the preset second data model. model; Wherein, the first data model and the second data model are used to represent constraint rules between data. 9. A monitoring and analysis system, characterized by comprising the data analysis system according to any one of claims 1 to 7, and at least one monitoring system. 10. A computer-readable storage medium, characterized by comprising program codes, which, when the program codes are run on a monitoring and analysis system, cause the monitoring and analysis system to perform the steps of the method of claim 8.