WO2020259155A1 - 一种生成告警数据报表的方法及装置 - Google Patents
一种生成告警数据报表的方法及装置 Download PDFInfo
- Publication number
- WO2020259155A1 WO2020259155A1 PCT/CN2020/091932 CN2020091932W WO2020259155A1 WO 2020259155 A1 WO2020259155 A1 WO 2020259155A1 CN 2020091932 W CN2020091932 W CN 2020091932W WO 2020259155 A1 WO2020259155 A1 WO 2020259155A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- alarm
- analysis task
- data frame
- report
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
Definitions
- the embodiment of the present invention relates to the technical field of financial technology (Fintech), in particular to a method and device for generating an alarm data report.
- Embodiments of the present invention provide a method and device for generating alarm data reports .
- an embodiment of the present invention provides a method for generating an alarm data report, including:
- the alarm data to be analyzed is obtained and the alarm data is used to construct a data frame.
- the alarm data is used to construct a data frame.
- For each alarm analysis task extract the partial data block corresponding to the alarm analysis task from the data frame, and then perform statistical analysis on the partial data block corresponding to the alarm analysis task, determine the result data block of the alarm analysis task, and then combine each alarm
- the result data blocks corresponding to the analysis tasks are spliced to determine the alarm data report. Therefore, only one alarm data report is generated for all alarm data to be analyzed.
- the alarm data report includes the content of all alarm analysis tasks, which is convenient for report management and user query reports. Secondly, processing alarm data in the form of data frames and generating type data corresponding to the report can improve the efficiency of generating alarm data reports.
- the splicing the result data blocks corresponding to each alarm analysis task to determine the alarm data report includes: adding the result data block corresponding to each alarm analysis task to the corresponding data frame group; All result data blocks in the data frame group are spliced to determine the result data frame corresponding to each alarm analysis task; the result data frame corresponding to each alarm analysis task is spliced to determine the alarm data report.
- the performing statistical analysis on the partial data blocks corresponding to the alarm analysis task to determine the result data blocks of the alarm analysis task includes: grouping and counting the partial data blocks corresponding to the alarm analysis task, Obtain multiple statistical result blocks; analyze and sort the multiple statistical result blocks, and generate the result data block of the alarm analysis task according to the top N analysis results, where N is a preset integer.
- the partial data blocks corresponding to the alarm analysis task are grouped and counted, which improves the statistical efficiency and makes the management and maintenance of the alarm data report more convenient.
- the data frame is a two-dimensional data structure including rows and columns.
- the method further includes: using a regular expression set from the data frame Extracting the data characteristic string from the mixed character string, and using the data characteristic string to replace the mixed character string of the data frame; performing type conversion on the data frame and assigning the missing value of the data frame.
- the regular expressions with different keywords but similar extraction feature string patterns are combined to construct a regular expression set.
- the regular expression set is used to extract the data feature string from the mixed character string of the data frame, and the data feature string is used to replace the mixed character string of the data frame, thereby improving the efficiency of feature string extraction and replacement.
- the data frame meets the requirements of sorting and statistics, which is convenient for subsequent alarm analysis based on the data frame.
- an embodiment of the present invention provides an apparatus for generating an alarm data report, including:
- An acquisition module configured to acquire alarm data to be analyzed and use the alarm data to construct a data frame
- the extraction module is used to extract the partial data block corresponding to the alarm analysis task from the data frame for each alarm analysis task;
- the analysis module is used to perform statistical analysis on the partial data blocks corresponding to the alarm analysis task, and determine the result data blocks of the alarm analysis task;
- the splicing module is used to splice the result data blocks corresponding to each alarm analysis task to determine the alarm data report.
- the splicing module is specifically configured to: add the result data block corresponding to each alarm analysis task to the corresponding data frame group; splice all the result data blocks in each data frame group to determine each The result data frame corresponding to the alarm analysis task; splicing the result data frame corresponding to each alarm analysis task to determine the alarm data report.
- the analysis module is specifically configured to: group and count the partial data blocks corresponding to the alarm analysis task to obtain multiple statistical result blocks; analyze and sort the multiple statistical result blocks, and according to The top N analysis results generate the result data block of the alarm analysis task, and N is a preset integer.
- the data frame is a two-dimensional data structure including rows and columns.
- the preprocessing module is specifically configured to: for each alarm analysis task, before extracting the partial data block corresponding to the alarm analysis task from the data frame corresponding to the alarm data, use The regular expression set extracts the data characteristic string from the mixed character string of the data frame, and replaces the mixed character string of the data frame with the data characteristic string; performs type conversion on the data frame and performs the type conversion on the data frame Assignment of missing values.
- an embodiment of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor.
- the processor generates an alarm data report when the program is executed. Steps of the method.
- an embodiment of the present invention provides a computer-readable storage medium that stores a computer program executable by a computer device, and when the program runs on the computer device, the computer device is executed to generate an alarm data report Steps of the method.
- FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present invention.
- FIG. 2 is a schematic flowchart of a method for generating an alarm data report provided by an embodiment of the present invention
- FIG. 3 is a schematic flowchart of a method for generating an alarm data report provided by an embodiment of the present invention
- FIG. 4 is a schematic structural diagram of an apparatus for generating an alarm data report provided by an embodiment of the present invention.
- Fig. 5 is a schematic structural diagram of a computer device provided by an embodiment of the present invention.
- Pandas A python data analysis package that provides the tools needed to efficiently manipulate large data sets.
- MySQL A relational database management system.
- CMDB configuration management database, configuration management database.
- the method for generating an alarm data report in the embodiment of the present invention can be applied to the application scenario shown in FIG. 1, and the application scenario includes a big data synchronization system 101, a source database 102, a report generation system 103, and a report database 104.
- the big data synchronization system 101 collects alarm data from a data source and writes the alarm data into the source database 102.
- the data source may be a business system of a financial institution such as a bank.
- the report generation system 103 generates an alarm data report based on the alarm data in the source database. Specifically, the report generation system 103 loads the alarm data to be analyzed from the source database and uses the alarm data to construct a data frame.
- each alarm analysis task For each alarm analysis task, extract the partial data block corresponding to the alarm analysis task from the data frame, and then perform statistical analysis on the partial data block corresponding to the alarm analysis task, determine the result data block of the alarm analysis task, and divide each alarm analysis task The corresponding result data blocks are spliced to determine the alarm data report. After that, the alarm data report is saved in the report database 104.
- the embodiment of the present invention provides a flow of a method for generating an alarm data report.
- the flow of the method can be executed by a device that generates an alarm data report.
- the device for generating an alarm data report can be
- the report generation system 103 shown in 1, as shown in Fig. 2, includes the following steps:
- Step S201 Obtain the alarm data to be analyzed and use the alarm data to construct a data frame.
- the big data synchronization system downloads alarm data from the data source and saves the alarm data in the source database.
- the alarm data in the source database can be updated according to the analysis period. For example, when the analysis period is one month, you can set the source database to be updated every month. When the analysis period is one week, you can set every Update the source database regularly every week. When the analysis cycle is one day, the source database can be updated regularly every day.
- the report generation system loads the alarm data to be analyzed from the source database.
- the data frame is a two-dimensional data structure including rows and columns. It can perform operations on the rows and columns of the data frame, and can also expand the rows and columns of the data frame.
- the pandas interface is used to load the alarm data to be analyzed into the memory from the source database, and then the alarm data is arranged in a row and column table into a two-dimensional data structure, thereby constructing a pandas data frame.
- Step S202 for each alarm analysis task, extract a partial data block corresponding to the alarm analysis task from the data frame.
- the alarm analysis task may be an analysis task for different fields, such as analyzing the loan field, analyzing the deposit field, analyzing the settlement field, and so on.
- Alarm analysis tasks can also be analysis tasks for different levels, such as analyzing alarm content, analyzing machines that frequently generate alarm content, and analyzing when the alarm content frequently occurs. Extract partial data blocks by column from the data frame, and the extracted partial data blocks correspond to the same alarm analysis task.
- the partial data blocks of each alarm analysis task can be configured in advance through parameterization, and the partial data blocks of each alarm analysis task may be different.
- Step S203 Perform statistical analysis on the partial data block corresponding to the alarm analysis task, and determine the result data block of the alarm analysis task.
- the partial data blocks corresponding to the alarm analysis task are grouped and counted to obtain multiple statistical result blocks; the multiple statistical result blocks are analyzed and sorted, according to the top N analysis results Generate the result data block of the alarm analysis task, where N is a preset integer.
- a pivot table can be used to perform grouping statistics on partial data blocks, and a cross table can be used to calculate the grouping frequency.
- the pivot table and the cross table are two operation methods for data blocks, which are both related and different.
- the connection is the cross table.
- a special pivot table can be replaced by a pivot table. The difference is that the cross table is dedicated to calculating the grouping frequency, while the pivot table is a function for grouping statistics, and the statistical type is specified by parameters.
- an analysis result block can be generated, where the statistical result block corresponds to the new column, and the analysis result block corresponds to the new column.
- the columns and new columns used in the necessary grouping are spliced to obtain the result data block .
- sorting you can sort from high to low, or from low to high.
- the partial data blocks are grouped according to time, and partial data blocks of the same analysis period are grouped into one group to obtain multiple grouped data blocks. Then, based on the alarm title field, count the number of occurrences of the same alarm content in each packet data block, and then add a new column to save the number of occurrences of each alarm content in each analysis cycle.
- the new columns corresponding to different analysis periods are spliced horizontally, and the increment and increase ratio of each alarm content are calculated through the spliced columns. Then add a new column corresponding to the increment and a new column corresponding to the increment ratio and splice it horizontally.
- the new partial data block can be extracted from the data frame based on the analysis result of the first N bits, and then the new partial data block can be grouped, counted, analyzed and combined Sort, take the new analysis results ranked in the top M, and generate a new result data block.
- the size of the new result data block is N ⁇ M, where N may be equal to M or not equal to M.
- step S204 the result data blocks corresponding to each alarm analysis task are spliced to determine an alarm data report.
- the result data block corresponds to the text, table, graph and other object types in the data report, and each result data block corresponds to a unique identifier.
- a large data frame can be obtained. Access the CMDB configuration system through the http(s) interface to update the data items in the big data frame.
- the pandas interface is used to convert big data frames into alarm data reports, and the alarm data reports are written into the report database.
- the big data frame can be updated by performing row and column operations on the big data frame.
- the alarm data is used to construct a data frame, and then partial data blocks in the data frame are extracted for alarm analysis to obtain the result data block of the alarm analysis task, and then the result data block corresponding to each alarm analysis task is spliced , Determine the alarm data report, so all the alarm data to be analyzed only generate an alarm data report, which includes the content of all alarm analysis tasks, which is convenient for report management and user query reports. Secondly, processing alarm data in the form of data frames and generating type data corresponding to the report can improve the efficiency of generating alarm data reports. By performing row and column operations on the big data frame, the alarm data report is updated, making the management and maintenance of the alarm data report more convenient.
- the embodiment of the present invention when generating the alarm data report, provides at least the following two implementation manners:
- the result data block corresponding to each alarm analysis task is marked with a type, and then added to the global data frame group. Then all result data blocks in the global data frame group are spliced to obtain a big data frame, and then the big data frame is converted into an alarm data report.
- all result data blocks in the global data frame group are vertically spliced, and each result data block uses the type column field to identify the type.
- the type column field When the type column field is different, it automatically expands horizontally and fills in missing values to form Big data frame.
- the number of rows of the obtained large data frame is the sum of the number of rows of all the result data blocks, and the number of columns is the union of the columns of all the result data blocks, that is, the same columns of the result data blocks are shared during splicing.
- each alarm analysis task can correspond to a data frame group, which is used to store the result data block of the alarm analysis task. Therefore, after the result data block corresponding to each alarm analysis task is generated , The result data block corresponding to each alarm analysis task can be added to the corresponding data frame group; then all the result data blocks in each data frame group are spliced to determine the result data frame corresponding to each alarm analysis task; The result data frame corresponding to each alarm analysis task is spliced to determine the alarm data report.
- all the result data blocks in the data frame group are vertically spliced to obtain the result data frame.
- the process of vertical splicing is the same as that in the foregoing embodiment, and will not be repeated here.
- all the result data frames are vertically spliced to obtain a big data frame, and then the big data frame is converted into an alarm data report.
- Save the result data blocks of different alarm analysis tasks by setting different data frame groups, first splicing the result data blocks in the data frame group to obtain the result data frame, and then splicing the result data frame to obtain the alarm data report, thereby improving the generation of alarm data The efficiency of the report.
- the data frame needs to be preprocessed before the above step S202, that is, before extracting the partial data block corresponding to the alarm analysis task from the data frame corresponding to the alarm data.
- the preprocessing method includes: using a regular expression set from The data characteristic string is extracted from the mixed character string of the data frame, and the mixed character string of the data frame is replaced with the data characteristic string, and the type conversion of the data frame is performed and the missing value of the data frame is assigned.
- the data frame preprocessing is performed by means of pandas matrix calculation.
- a regular expression corresponding to a keyword to extract the data feature string it takes about 0.8-1s due to the large amount of data. If you set more keywords, the sequential processing time will increase linearly. For example, 5 keywords will take about 4-5s. If the regular expressions corresponding to these 5 keywords are merged into one regular expression, it takes about the same time as executing a regular expression. Therefore, in the embodiment of the present invention, regular expressions with different keywords but similar extraction feature string patterns are combined to construct a regular expression set.
- the regular expression set is used to extract the data feature string from the mixed character string of the data frame, and the data feature string is used to replace the mixed character string of the data frame, thereby improving the efficiency of feature string extraction and replacement.
- the data frame meets the requirements of sorting and statistics, which is convenient for subsequent alarm analysis based on the data frame.
- the following describes a method for generating alarm data reports provided by the embodiments of the present invention in combination with specific implementation scenarios.
- the method is interacted by a big data synchronization system, a source database, a report generation system, and a report database. Execution, as shown in Figure 3, the method includes the following steps:
- the big data synchronization system collects alarm data from the data source and writes the alarm data into the source database.
- the data source may be a business system of a financial institution such as a bank.
- the alarm data in the source database is updated regularly.
- the report generation system includes a big data loading module, a preprocessing module, a report data generation module, a data frame update module, and a report storage module.
- the big data loading module uses the pandas interface to load the alarm data to be analyzed into the memory from the source database, and then uses the alarm data to construct a pandas data frame.
- the preprocessing module uses pandas matrix calculation to perform data frame preprocessing.
- the preprocessing includes data feature string extraction, data type conversion, and missing value processing.
- the report data generation module extracts the partial data blocks corresponding to the alarm analysis task from the data frame, groups and counts the partial data blocks corresponding to the alarm analysis task, and obtains multiple statistical result blocks. Splice multiple result data blocks horizontally, and analyze multiple statistical result blocks to obtain analysis result blocks, sort the analysis results in each analysis result block, and take the top N analysis in each analysis result block As a result, the result data block of the alarm analysis task is generated based on the alarm analysis task, the statistical result block, and the analysis result block after sorting the value operation, and N is a preset integer. Add the result data block to the global data frame group after marking the type. Determine whether to continue extracting local data blocks.
- the data block uses the type column field to identify the type. When the type column field is different, it automatically expands horizontally and fills in missing values to form a big data frame.
- the data frame update module accesses the CMDB configuration system through the http(s) interface to update the data items in the big data frame.
- the report storage module uses the pandas data writing interface to convert big data frames into alarm data reports, and writes the alarm data reports into the report database.
- the alarm data is used to construct a data frame, and then partial data blocks in the data frame are extracted for alarm analysis to obtain the result data block of the alarm analysis task, and then the result data block of the same alarm analysis task is spliced to obtain the result Data frame, the result data frames of all alarm analysis tasks are spliced to obtain an alarm data report, so all alarm data to be analyzed only generates an alarm data report, which includes the content of all alarm analysis tasks, which is convenient for report management and User query report.
- processing alarm data in the form of data frames and generating type data corresponding to the report can improve the efficiency of generating alarm data reports.
- the alarm data report is updated, making the management and maintenance of the alarm data report more convenient.
- an embodiment of the present invention provides a schematic structural diagram of an apparatus for generating an alarm data report.
- the apparatus 400 includes:
- the obtaining module 401 is configured to obtain alarm data to be analyzed and use the alarm data to construct a data frame;
- the extraction module 402 is configured to extract a partial data block corresponding to the alarm analysis task from the data frame for each alarm analysis task;
- the analysis module 403 is configured to perform statistical analysis on the partial data blocks corresponding to the alarm analysis task, and determine the result data blocks of the alarm analysis task;
- the splicing module 404 is used for splicing the result data blocks corresponding to each alarm analysis task to determine the alarm data report.
- the splicing module 404 is specifically configured to:
- the result data frame corresponding to each alarm analysis task is spliced to determine the alarm data report.
- the analysis module 403 is specifically configured to:
- the multiple statistical result blocks are analyzed and sorted, and the result data block of the alarm analysis task is generated according to the top N analysis results, where N is a preset integer.
- the data frame is a two-dimensional data structure including rows and columns.
- it further includes a preprocessing module 405;
- the preprocessing module 405 is specifically configured to:
- a regular expression set is used to extract the data characteristic string from the mixed character string of the data frame, And replace the mixed character string of the data frame with the data characteristic string;
- an embodiment of the present invention provides a computer device. As shown in FIG. 5, it includes at least one processor 501 and a memory 502 connected to the at least one processor.
- the embodiment of the present invention does not limit the processor.
- the connection between the processor 501 and the memory 502 in FIG. 5 is taken as an example.
- the bus can be divided into address bus, data bus, control bus, etc.
- the memory 502 stores instructions that can be executed by at least one processor 501.
- the at least one processor 501 can execute the steps included in the aforementioned method for generating alarm data reports. .
- the processor 501 is the control center of the computer equipment, which can use various interfaces and lines to connect to various parts of the computer equipment, and generate an alarm by running or executing instructions stored in the memory 502 and calling data stored in the memory 502 data report.
- the processor 501 may include one or more processing units, and the processor 501 may integrate an application processor and a modem processor.
- the application processor mainly processes the operating system, user interface, and application programs.
- the adjustment processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 501.
- the processor 501 and the memory 502 may be implemented on the same chip, and in some embodiments, they may also be implemented on separate chips.
- the processor 501 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application specific integrated circuit (ASIC), a field programmable gate array or other programmable logic devices, discrete gates or transistors Logic devices and discrete hardware components can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention.
- the general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present invention may be directly embodied as being executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.
- the memory 502 can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules.
- the memory 502 may include at least one type of storage medium, for example, it may include flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), magnetic memory, disk , CD, etc.
- the memory 502 is any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.
- the memory 502 in the embodiment of the present invention may also be a circuit or any other device capable of realizing a storage function for storing program instructions and/or data.
- the embodiments of the present invention provide a computer-readable storage medium that stores a computer program executable by a computer device, and when the program runs on the computer device, the computer device is executed to generate an alarm. The steps of the data report method.
- the embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
- a computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
- These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
- the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
- These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
- the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Fuzzy Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Alarm Systems (AREA)
- Maintenance And Management Of Digital Transmission (AREA)
Abstract
一种生成告警数据报表的方法及装置,涉及金融科技技术领域,方法包括:获取待分析的告警数据并采用告警数据构造数据帧(201);针对每个告警分析任务,从数据帧中抽取告警分析任务对应的局部数据块(202);对告警分析任务对应的局部数据块进行统计分析,确定告警分析任务的结果数据块(203);将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表(204)。故所有待分析的告警数据仅生成一张告警数据报表,该告警数据报表包括所有告警分析任务的内容,便于报表管理以及用户查询报表。以数据帧的形式处理告警数据,生成报表对应的类型数据,能提高生成告警数据报表的效率。通过对大数据帧进行行和列的运算,实现对告警数据报表的更新,使得告警数据报表的管理维护更方便。
Description
相关申请的交叉引用
本申请要求在2019年06月27日提交中国专利局、申请号为201910569785.7、申请名称为“一种生成告警数据报表的方法及装置”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本发明实施例涉及金融科技(Fintech)技术领域,尤其涉及一种生成告警数据报表的方法及装置。
随着计算机技术的发展,越来越多的技术应用在金融领域,传统金融业正在逐步向金融科技(Fintech)转变,但由于金融行业的安全性、实时性要求,也对技术提出了更高的要求。目前,在对金融系统中各个机器的性能指标以及服务软件的健康状态进行告警分析时,采集的原始数据进行处理后保存在大数据平台集市中。由于在大数据平台的数据集市中,涉及的数据源表会很多,故采用结构化查询语言(structured query language,SQL)语句从数据集市不同区域提取数据时,配置报表不同区域取数的SQL语句特别复杂,不便于数据查询。
发明内容
由于大数据集市中涉及的数据源表很多,查询语句需要从不同区域提取数据,导致查询语句复杂,不便于数据查询的问题,本发明实施例提供了一种生成告警数据报表的方法及装置。
第一方面,本发明实施例提供了一种生成告警数据报表的方法,包括:
获取待分析的告警数据并采用所述告警数据构造数据帧;针对每个告警 分析任务,从所述数据帧中抽取所述告警分析任务对应的局部数据块;对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块;将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。
本发明实施例中,获取待分析的告警数据并采用告警数据构造数据帧。针对每个告警分析任务,从数据帧中抽取告警分析任务对应的局部数据块,然后对告警分析任务对应的局部数据块进行统计分析,确定告警分析任务的结果数据块,之后再将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。故所有待分析的告警数据仅生成一张告警数据报表,该告警数据报表包括所有告警分析任务的内容,便于报表管理以及用户查询报表。其次,以数据帧的形式处理告警数据,生成报表对应的类型数据,能提高生成告警数据报表的效率。
可选地,所述将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表,包括:将每个告警分析任务对应的结果数据块添加至对应的数据帧组中;将每个数据帧组中的所有结果数据块进行拼接,确定每个告警分析任务对应的结果数据帧;将每个告警分析任务对应的结果数据帧进行拼接,确定告警数据报表。
通过上述方法,通过设置不同的数据帧组分别保存不同告警分析任务的结果数据块,先拼接数据帧组内的结果数据块获得结果数据帧,然后将结果数据帧进行拼接获得告警数据报表,从而提高生成告警数据报表的效率。
可选地,所述对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块,包括:对所述告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块;对所述多个统计结果块进行分析并排序,并根据排在前N位的分析结果生成所述告警分析任务的结果数据块,N为预设整数。
通过上述方法,对所述告警分析任务对应的局部数据块进行分组并统计,提高了统计效率,使得告警数据报表的管理维护更方便。
可选地,所述数据帧是包括行和列的二维数据结构。
可选地,所述针对每个告警分析任务,从所述告警数据对应的数据帧中抽取所述告警分析任务对应的局部数据块之前,还包括:采用正则表达式集合从所述数据帧的混合字符串中提取数据特征串,并采用所述数据特征串替换所述数据帧的混合字符串;对所述数据帧进行类型转换以及对所述数据帧的缺失值赋值。
通过上述方法,将关键字不同但提取特征串模式相近的正则表达式合并,构造正则表达式集合。然后采用正则表达式集合从数据帧的混合字符串中提取数据特征串,并采用数据特征串替换数据帧的混合字符串,从而提高特征串提取和替换的效率。通过数据帧列类型转换对特定列执行日期时间类型或整数类型转换,通过pandas接口对数据帧按条件做缺失值处理以及对数据帧行按条件赋值。通过对数据帧进行预处理,使得数据帧满足排序和统计的要求,便于后续基于数据帧进行告警分析。
第二方面,本发明实施例提供了一种生成告警数据报表的装置,包括:
获取模块,用于获取待分析的告警数据并采用所述告警数据构造数据帧;
抽取模块,用于针对每个告警分析任务,从所述数据帧中抽取所述告警分析任务对应的局部数据块;
分析模块,用于对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块;
拼接模块,用于将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。
可选地,所述拼接模块具体用于:将每个告警分析任务对应的结果数据块添加至对应的数据帧组中;将每个数据帧组中的所有结果数据块进行拼接,确定每个告警分析任务对应的结果数据帧;将每个告警分析任务对应的结果数据帧进行拼接,确定告警数据报表。
可选地,所述分析模块具体用于:对所述告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块;对所述多个统计结果块进行分析 并排序,并根据排在前N位的分析结果生成所述告警分析任务的结果数据块,N为预设整数。
可选地,所述数据帧是包括行和列的二维数据结构。
可选地,还包括预处理模块;所述预处理模块具体用于:针对每个告警分析任务,从所述告警数据对应的数据帧中抽取所述告警分析任务对应的局部数据块之前,采用正则表达式集合从所述数据帧的混合字符串中提取数据特征串,并采用所述数据特征串替换所述数据帧的混合字符串;对所述数据帧进行类型转换以及对所述数据帧的缺失值赋值。
第三方面,本发明实施例提供了一种计算机设备,包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,所述处理器执行所述程序时实现生成告警数据报表的方法的步骤。
第四方面,本发明实施例提供了一种计算机可读存储介质,其存储有可由计算机设备执行的计算机程序,当所述程序在计算机设备上运行时,使得所述计算机设备执行生成告警数据报表的方法的步骤。
图1为本发明实施例提供的一种应用场景示意图;
图2为本发明实施例提供的一种生成告警数据报表的方法的流程示意图;
图3为本发明实施例提供的一种生成告警数据报表的方法的流程示意图;
图4为本发明实施例提供的一种生成告警数据报表的装置的结构示意图;
图5为本发明实施例提供的一种计算机设备的结构示意图。
为了方便理解,下面对本发明实施例中涉及的名词进行解释。
Pandas:一个python数据分析包,提供高效地操作大型数据集所需的工具。
MySQL:一个关系型数据库管理系统。
CMDB:configuration management database,配置管理数据库。
本发明实施例中的生成告警数据报表的方法可以应用于如图1所示的应用场景,在该应用场景中包括大数据同步系统101、源数据库102、报表生成系统103、报表数据库104。大数据同步系统101从数据源采集告警数据并将告警数据写入源数据库102,数据源可以是银行等金融机构的业务系统。报表生成系统103基于源数据库中的告警数据生成告警数据报表。具体地,报表生成系统103从源数据库中加载待分析的告警数据并采用告警数据构造数据帧。针对每个告警分析任务,从数据帧中抽取告警分析任务对应的局部数据块,然后对告警分析任务对应的局部数据块进行统计分析,确定告警分析任务的结果数据块,将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。之后再将告警数据报表保存在报表数据库104中。
基于图1所示的应用场景图,本发明实施例提供了一种生成告警数据报表的方法的流程,该方法的流程可以由生成告警数据报表的装置执行,生成告警数据报表的装置可以是图1所示的报表生成系统103,如图2所示,包括以下步骤:
步骤S201,获取待分析的告警数据并采用告警数据构造数据帧。
具体地,大数据同步系统从数据源中下载告警数据并将告警数据保存在源数据库中。为了便于对告警数据进行分析,可以根据分析周期更新源数据库中的告警数据,比如,分析周期为一个月时,可以设定每个月定期更新源数据库,分析周期为一周时,可以设定每周定期更新源数据库。分析周期为一天时,可以设定每天定期更新源数据库。
报表生成系统从源数据库中加载待分析的告警数据,数据帧是包括行和列的二维数据结构,可以对数据帧的行和列进行运算,也可以扩展数据帧的行和列。具体实施中,使用pandas接口从源数据库将待分析的告警数据载入内存,然后将告警数据以行和列的表格方式排列成二维数据结构,从而构造出pandas数据帧。由于加载数据量大,故首先可以筛选特征字段,仅载入数据分析必需字段,减少加载数据规模;其次,可以通过多进程/多线程并行加载 技术,加快加载告警数据的速度;最后,载入内存的告警数据压缩存放,用pandas接口将告警数据构造成数据帧,降低加载数据耗时。
步骤S202,针对每个告警分析任务,从数据帧中抽取告警分析任务对应的局部数据块。
具体地,告警分析任务可以是针对不同领域的分析任务,比如分析贷款领域、分析存款领域、分析结算领域等。告警分析任务也可以是针对不同层级的分析任务,比如分析告警内容、分析告警内容频发的机器、分析告警内容频发的时间等。从数据帧中按列抽取局部数据块,抽取的局部数据块对应同一个告警分析任务。具体实施中,预先可以通过参数化配置每个告警分析任务的局部数据块,每个告警分析任务的局部数据块可以是不相同的。
步骤S203,对告警分析任务对应的局部数据块进行统计分析,确定告警分析任务的结果数据块。
在一种可能的实施方式中,对告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块;对多个统计结果块进行分析并排序,根据排在前N位的分析结果生成所述告警分析任务的结果数据块,N为预设整数。
具体实施中,可以采用透视表对局部数据块进行分组统计,采用交叉表计算分组频率,透视表和交叉表是对数据块的两种操作方式,既有联系又有区别,联系是交叉表是一种特殊的透视表,可以用透视表代替。区别是交叉表专用于计算分组频率,而透视表是一种进行分组统计的函数,通过参数指定统计类型。
在对统计结果块进行分析后,可以生成分析结果块,其中,统计结果块对应新列,分析结果块对应新列,将必要的分组时所用到的列以及新列进行拼接,获得结果数据块。排序时可以按照从高到低排序,也可以从低到高排序。
示例性地,按照时间对局部数据块进行分组,将相同分析周期的局部数据块分为一组,获得多个分组数据块。然后基于告警标题字段统计各个分组数据块中相同告警内容的出现次数,然后新增列保存每个分析周期中的各告警内容的出现次数。将不同分析周期对应的新列横向拼接,通过拼接后的列 统计各告警内容的增量、增比。然后新增增量对应的新列、增比对应的新列并横向拼接,按照增量从高到低的顺序对新列中的增量值进行排序,取新列中的前N位增量值。按照增比从高到低的顺序对新列中的增比值进行排序,取新列中的前N位增比值,之后再基于新列生成结果数据块并标注结果数据块的类型。
需要说明的是,当需要继续对前N位的分析结果继续进行统计分析时,比如,在获知告警内容的数量、告警内容的增量、告警内容的增比的情况下,需要定位告警频发的机器、告警频发的日期、告警频发的对象时,可以基于前N位的分析结果重新从数据帧中抽取新的局部数据块,然后对新的局部数据块进行分组、统计、分析并排序,取排在前M位的新分析结果,生成新的结果数据块,新的结果数据块的规模是N×M,其中,N可以等于M,也可以不等于M。
步骤S204,将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。
结果数据块与数据报表中的文本、表格、图形等对象类型对应,每个结果数据块对应唯一标识。将每个告警分析任务对应的结果数据块进行拼接后,可以获得一个大数据帧。通过http(s)接口访问CMDB配置系统,对大数据帧内的数据项进行更新。采用pandas接口将大数据帧转化为告警数据报表,并将告警数据报表写入报表数据库中。通过对大数据帧进行行和列的运算可以更新大数据帧,当新增告警分析任务时,大数据帧中新增不同类型的结果数据块,进而在大数据帧中新增列字段,从而使大数据帧的存储结构发生改变。
本发明实施例中,采用告警数据构造数据帧,然后抽取数据帧中的局部数据块进行告警分析,获得告警分析任务的结果数据块,之后再将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表,故所有待分析的告警数据仅生成一张告警数据报表,该告警数据报表包括所有告警分析任务的内容,便于报表管理以及用户查询报表。其次,以数据帧的形式处理告警数据,生成报表对应的类型数据,能提高生成告警数据报表的效率。通过对 大数据帧进行行和列的运算,实现对告警数据报表的更新,使得告警数据报表的管理维护更方便。
可选地,在上述步骤S204中,在生成告警数据报表时,本发明实施例至少提供以下两种实施方式:
在一种可能的实施方式中,将每个告警分析任务对应的结果数据块标注类型后,添加至全局数据帧组中。然后将全局数据帧组中的所有结果数据块进行拼接,获得大数据帧,之后再将大数据帧转化为告警数据报表。
具体实施中,将全局数据帧组中的所有结果数据块进行纵向拼接,每个结果数据块采用类型列字段标识类型,当类型列字段不同时,自动横向扩展,同时对缺失值进行填充,形成大数据帧。获得的大数据帧的行数量是所有结果数据块的行数之和,列数量是所有结果数据块的列的并集,即拼接时结果数据块相同的列共用。
在一种可能的实施方式中,每个告警分析任务可以对应一个数据帧组,该数据帧组用于保存告警分析任务的结果数据块,故在生成每个告警分析任务对应的结果数据块后,可以将每个告警分析任务对应的结果数据块添加至对应的数据帧组中;然后将每个数据帧组中的所有结果数据块进行拼接,确定每个告警分析任务对应的结果数据帧;将每个告警分析任务对应的结果数据帧进行拼接,确定告警数据报表。
具体实施中,将数据帧组中的所有结果数据块进行纵向拼接,获得结果数据帧,纵向拼接的过程与前述实施方式中的过程相同,此处不再赘述。然后将所有结果数据帧进行纵向拼接,获得大数据帧,之后再将大数据帧转化为告警数据报表。通过设置不同的数据帧组分别保存不同告警分析任务的结果数据块,先拼接数据帧组内的结果数据块获得结果数据帧,然后将结果数据帧进行拼接获得告警数据报表,从而提高生成告警数据报表的效率。
可选地,在上述步骤S202之前,即从告警数据对应的数据帧中抽取告警分析任务对应的局部数据块之前,需要对数据帧进行预处理,预处理的方式包括:采用正则表达式集合从数据帧的混合字符串中提取数据特征串,并采 用数据特征串替换数据帧的混合字符串,对数据帧进行类型转换以及对数据帧的缺失值赋值。
具体实施中,采用pandas矩阵计算的方式执行数据帧预处理。当按照一个关键字对应执行一条正则表达式提取数据特征串时,由于数据量大,耗时为0.8-1s左右。若设置的关键字越多,顺序处理耗时将呈线性增长,比如,5个关键字耗时在4-5s左右。如果将这5个关键字对应的正则表达式合并成一个正则表达式时,耗时与执行一条正则表达式差不多。因此,本发明实施例中将关键字不同但提取特征串模式相近的正则表达式合并,构造正则表达式集合。然后采用正则表达式集合从数据帧的混合字符串中提取数据特征串,并采用数据特征串替换数据帧的混合字符串,从而提高特征串提取和替换的效率。通过数据帧列类型转换对特定列执行日期时间类型或整数类型转换,通过pandas接口对数据帧按条件做缺失值处理以及对数据帧行按条件赋值。通过对数据帧进行预处理,使得数据帧满足排序和统计的要求,便于后续基于数据帧进行告警分析。
为了更好的解释本发明实施例,下面结合具体的实施场景描述本发明实施例提供的一种生成告警数据报表的方法,该方法由大数据同步系统、源数据库、报表生成系统、报表数据库交互执行,如图3所示,该方法包括以下步骤:
大数据同步系统从数据源采集告警数据并将告警数据写入源数据库,数据源可以是银行等金融机构的业务系统。源数据库中的告警数据定期更新。报表生成系统包括大数据加载模块、预处理模块、报表数据生成模块、数据帧更新模块、报表存储模块。大数据加载模块使用pandas接口从源数据库将待分析的告警数据载入内存,然后采用告警数据构造pandas数据帧。预处理模块采用pandas矩阵计算的方式执行数据帧预处理,预处理包括数据特征串提取、数据类型转换、缺失值处理。报表数据生成模块从数据帧中抽取告警分析任务对应的局部数据块,对告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块。将多个结果数据块进行横向拼接,并多个统计 结果块进行分析获得分析结果块,对每个分析结果块中的分析结果进行排序,取每个分析结果块中排在前N位的分析结果,基于告警分析任务、统计结果块、排序后进行取值操作的分析结果块生成告警分析任务的结果数据块,N为预设整数。将结果数据块标注类型后添加至全局数据帧组中。判断是否继续抽取局部数据块,若是,执行上述抽取数据块、分组、统计、分析、排序和生成结果数据块的步骤,否则,将全局数据帧组中的结果数据块进行纵向拼接,每个结果数据块采用类型列字段标识类型,当类型列字段不同时,自动横向扩展,同时对缺失值进行填充,形成大数据帧。数据帧更新模块通过http(s)接口访问CMDB配置系统,对大数据帧内的数据项进行更新。报表存储模块采用pandas写数据接口将大数据帧转化为告警数据报表,并将告警数据报表写入报表数据库中。
本发明实施例中,采用告警数据构造数据帧,然后抽取数据帧中的局部数据块进行告警分析,获得告警分析任务的结果数据块,之后再将同一告警分析任务的结果数据块进行拼接获得结果数据帧,将所有告警分析任务的结果数据帧进行拼接获得告警数据报表,故所有待分析的告警数据仅生成一张告警数据报表,该告警数据报表包括所有告警分析任务的内容,便于报表管理以及用户查询报表。其次,以数据帧的形式处理告警数据,生成报表对应的类型数据,能提高生成告警数据报表的效率。通过对大数据帧进行行和列的运算,实现对告警数据报表的更新,使得告警数据报表的管理维护更方便。
基于相同的技术构思,本发明实施例提供了一种生成告警数据报表的装置的结构示意图,如图4所示,该装置400包括:
获取模块401,用于获取待分析的告警数据并采用所述告警数据构造数据帧;
抽取模块402,用于针对每个告警分析任务,从所述数据帧中抽取所述告警分析任务对应的局部数据块;
分析模块403,用于对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块;
拼接模块404,用于将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。
可选地,所述拼接模块404具体用于:
将每个告警分析任务对应的结果数据块添加至对应的数据帧组中;
将每个数据帧组中的所有结果数据块进行拼接,确定每个告警分析任务对应的结果数据帧;
将每个告警分析任务对应的结果数据帧进行拼接,确定告警数据报表。
可选地,所述分析模块403具体用于:
对所述告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块;
对所述多个统计结果块进行分析并排序,根据排在前N位的分析结果生成所述告警分析任务的结果数据块,N为预设整数。
可选地,所述数据帧是包括行和列的二维数据结构。
可选地,还包括预处理模块405;
所述预处理模块405具体用于:
针对每个告警分析任务,从所述告警数据对应的数据帧中抽取所述告警分析任务对应的局部数据块之前,采用正则表达式集合从所述数据帧的混合字符串中提取数据特征串,并采用所述数据特征串替换所述数据帧的混合字符串;
对所述数据帧进行类型转换以及对所述数据帧的缺失值赋值。
基于相同的技术构思,本发明实施例提供了一种计算机设备,如图5所示,包括至少一个处理器501,以及与至少一个处理器连接的存储器502,本发明实施例中不限定处理器501与存储器502之间的具体连接介质,图5中处理器501和存储器502之间通过总线连接为例。总线可以分为地址总线、数据总线、控制总线等。
在本发明实施例中,存储器502存储有可被至少一个处理器501执行的指令,至少一个处理器501通过执行存储器502存储的指令,可以执行前述 的生成告警数据报表的方法中所包括的步骤。
其中,处理器501是计算机设备的控制中心,可以利用各种接口和线路连接计算机设备的各个部分,通过运行或执行存储在存储器502内的指令以及调用存储在存储器502内的数据,从而生成告警数据报表。可选的,处理器501可包括一个或多个处理单元,处理器501可集成应用处理器和调制解调处理器,其中,应用处理器主要处理操作系统、用户界面和应用程序等,调制解调处理器主要处理无线通信。可以理解的是,上述调制解调处理器也可以不集成到处理器501中。在一些实施例中,处理器501和存储器502可以在同一芯片上实现,在一些实施例中,它们也可以在独立的芯片上分别实现。
处理器501可以是通用处理器,例如中央处理器(CPU)、数字信号处理器、专用集成电路(Application Specific Integrated Circuit,ASIC)、现场可编程门阵列或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件,可以实现或者执行本发明实施例中公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本发明实施例所公开的方法的步骤可以直接体现为硬件处理器执行完成,或者用处理器中的硬件及软件模块组合执行完成。
存储器502作为一种非易失性计算机可读存储介质,可用于存储非易失性软件程序、非易失性计算机可执行程序以及模块。存储器502可以包括至少一种类型的存储介质,例如可以包括闪存、硬盘、多媒体卡、卡型存储器、随机访问存储器(Random Access Memory,RAM)、静态随机访问存储器(Static Random Access Memory,SRAM)、可编程只读存储器(Programmable Read Only Memory,PROM)、只读存储器(Read Only Memory,ROM)、带电可擦除可编程只读存储器(Electrically Erasable Programmable Read-Only Memory,EEPROM)、磁性存储器、磁盘、光盘等等。存储器502是能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。本发明实施例中的存储器502还可以是电路或者其它 任意能够实现存储功能的装置,用于存储程序指令和/或数据。
基于相同的技术构思,本发明实施例提供了一种计算机可读存储介质,其存储有可由计算机设备执行的计算机程序,当所述程序在计算机设备上运行时,使得所述计算机设备执行生成告警数据报表的方法的步骤。
本领域内的技术人员应明白,本发明的实施例可提供为方法、或计算机程序产品。因此,本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且,本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器,使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中,使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品,该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上,使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理,从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
尽管已描述了本发明的优选实施例,但本领域内的技术人员一旦得知了基本创造性概念,则可对这些实施例作出另外的变更和修改。所以,所附权 利要求意欲解释为包括优选实施例以及落入本发明范围的所有变更和修改。
显然,本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样,倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内,则本发明也意图包含这些改动和变型在内。
Claims (10)
- 一种生成告警数据报表的方法,其特征在于,包括:获取待分析的告警数据并采用所述告警数据构造数据帧;针对每个告警分析任务,从所述数据帧中抽取所述告警分析任务对应的局部数据块;对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块;将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。
- 如权利要求1所述的方法,其特征在于,所述将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表,包括:将每个告警分析任务对应的结果数据块添加至对应的数据帧组中;将每个数据帧组中的所有结果数据块进行拼接,确定每个告警分析任务对应的结果数据帧;将每个告警分析任务对应的结果数据帧进行拼接,确定告警数据报表。
- 如权利要求1所述的方法,其特征在于,所述对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块,包括:对所述告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块;对所述多个统计结果块进行分析并排序,根据排在前N位的分析结果生成所述告警分析任务的结果数据块,N为预设整数。
- 如权利要求1所述的方法,其特征在于,所述数据帧是包括行和列的二维数据结构。
- 如权利要求1至4任一所述的方法,其特征在于,所述针对每个告警分析任务,从所述告警数据对应的数据帧中抽取所述告警分析任务对应的局部数据块之前,还包括:采用正则表达式集合从所述数据帧的混合字符串中提取数据特征串,并 采用所述数据特征串替换所述数据帧的混合字符串;对所述数据帧进行类型转换以及对所述数据帧的缺失值赋值。
- 一种生成告警数据报表的装置,其特征在于,包括:获取模块,用于获取待分析的告警数据并采用所述告警数据构造数据帧;抽取模块,用于针对每个告警分析任务,从所述数据帧中抽取所述告警分析任务对应的局部数据块;分析模块,用于对所述告警分析任务对应的局部数据块进行统计分析,确定所述告警分析任务的结果数据块;拼接模块,用于将每个告警分析任务对应的结果数据块进行拼接,确定告警数据报表。
- 如权利要求6所述的装置,其特征在于,所述拼接模块具体用于:将每个告警分析任务对应的结果数据块添加至对应的数据帧组中;将每个数据帧组中的所有结果数据块进行拼接,确定每个告警分析任务对应的结果数据帧;将每个告警分析任务对应的结果数据帧进行拼接,确定告警数据报表。
- 如权利要求6所述的装置,其特征在于,所述分析模块具体用于:对所述告警分析任务对应的局部数据块进行分组并统计,获得多个统计结果块;对所述多个统计结果块进行分析并排序,并根据排在前N位的分析结果生成所述告警分析任务的结果数据块,N为预设整数。
- 一种计算机设备,包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序,其特征在于,所述处理器执行所述程序时实现权利要求1~5任一权利要求所述方法的步骤。
- 一种计算机可读存储介质,其特征在于,其存储有可由计算机设备执行的计算机程序,当所述程序在计算机设备上运行时,使得所述计算机设备执行权利要求1~5任一所述方法的步骤。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201910569785.7 | 2019-06-27 | ||
CN201910569785.7A CN110287241B (zh) | 2019-06-27 | 2019-06-27 | 一种生成告警数据报表的方法及装置 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020259155A1 true WO2020259155A1 (zh) | 2020-12-30 |
Family
ID=68019964
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2020/091932 WO2020259155A1 (zh) | 2019-06-27 | 2020-05-22 | 一种生成告警数据报表的方法及装置 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN110287241B (zh) |
WO (1) | WO2020259155A1 (zh) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110287241B (zh) * | 2019-06-27 | 2023-09-08 | 深圳前海微众银行股份有限公司 | 一种生成告警数据报表的方法及装置 |
CN113204416A (zh) * | 2021-04-07 | 2021-08-03 | 上海多维度网络科技股份有限公司 | 数据报表任务执行方法、装置、设备和存储介质 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101170729A (zh) * | 2007-11-30 | 2008-04-30 | 中国移动通信集团重庆有限公司 | 一种gsm网络预警分析系统及网络预警分析方法 |
US20090300044A1 (en) * | 2008-05-27 | 2009-12-03 | Zheng Jerry | Systems and methods for automatically identifying data dependencies for reports |
CN102841943A (zh) * | 2012-08-24 | 2012-12-26 | 上海泰宇信息技术有限公司 | 数据安全监管预警及备份策略系统及方法 |
CN110287241A (zh) * | 2019-06-27 | 2019-09-27 | 深圳前海微众银行股份有限公司 | 一种生成告警数据报表的方法及装置 |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170187737A1 (en) * | 2015-12-28 | 2017-06-29 | Le Holdings (Beijing) Co., Ltd. | Method and electronic device for processing user behavior data |
CN108763038B (zh) * | 2018-08-08 | 2022-04-12 | 平安科技(深圳)有限公司 | 告警数据的管理方法、装置、计算机设备及存储介质 |
CN109933771B (zh) * | 2019-03-22 | 2023-04-14 | 广州市玄武无线科技股份有限公司 | 一种报表自动合并方法、装置、设备及存储介质 |
-
2019
- 2019-06-27 CN CN201910569785.7A patent/CN110287241B/zh active Active
-
2020
- 2020-05-22 WO PCT/CN2020/091932 patent/WO2020259155A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101170729A (zh) * | 2007-11-30 | 2008-04-30 | 中国移动通信集团重庆有限公司 | 一种gsm网络预警分析系统及网络预警分析方法 |
US20090300044A1 (en) * | 2008-05-27 | 2009-12-03 | Zheng Jerry | Systems and methods for automatically identifying data dependencies for reports |
CN102841943A (zh) * | 2012-08-24 | 2012-12-26 | 上海泰宇信息技术有限公司 | 数据安全监管预警及备份策略系统及方法 |
CN110287241A (zh) * | 2019-06-27 | 2019-09-27 | 深圳前海微众银行股份有限公司 | 一种生成告警数据报表的方法及装置 |
Also Published As
Publication number | Publication date |
---|---|
CN110287241A (zh) | 2019-09-27 |
CN110287241B (zh) | 2023-09-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11762882B2 (en) | System and method for analysis and management of data distribution in a distributed database environment | |
CN111538731B (zh) | 一种工业数据自动生成报表系统 | |
US20130227194A1 (en) | Active non-volatile memory post-processing | |
US20150032708A1 (en) | Database analysis apparatus and method | |
WO2020259155A1 (zh) | 一种生成告警数据报表的方法及装置 | |
CN104778540A (zh) | 一种建材装备制造bom管理方法及管理系统 | |
US20160283610A1 (en) | Hybrid flows containing a continous flow | |
CN106780149A (zh) | 一种基于定时任务调度的设备实时监测系统 | |
CN112883042A (zh) | 数据更新及展示方法、装置、电子设备及存储介质 | |
CN104317942A (zh) | 一种基于Hadoop云平台的海量数据比对方法及系统 | |
CN109669975B (zh) | 一种工业大数据处理系统及方法 | |
US20140325405A1 (en) | Auto-completion of partial line pattern | |
CN111367951A (zh) | 一种流数据处理的方法及装置 | |
CN110908870B (zh) | 一种大型机的资源监控方法、装置、存储介质及设备 | |
CN104331335A (zh) | 门户网站的死链检查方法和装置 | |
CN103064991A (zh) | 一种海量数据聚类方法 | |
CN107844490A (zh) | 一种数据库的分库方法及装置 | |
US10042902B2 (en) | Business rules influenced quasi-cubes with higher diligence of data optimization | |
CN115470279A (zh) | 基于企业数据的数源转换方法、装置、设备及介质 | |
CN115905113A (zh) | 一种生成数据快照的方法及装置 | |
CN115757481A (zh) | 数据迁移方法、装置、设备和存储介质 | |
CN112818655B (zh) | 一种基于模板和文件追加写的excel数据处理方法和工具 | |
CN114116773A (zh) | 一种结构化查询语言sql文本审核方法及装置 | |
CN113344023A (zh) | 一种代码推荐方法、装置及系统 | |
US20150081735A1 (en) | System and method for fast identification of variable roles during initial data exploration |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20831721 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 22.04.2022) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20831721 Country of ref document: EP Kind code of ref document: A1 |