CN115237724A

CN115237724A - Data monitoring method, device, equipment and storage medium based on artificial intelligence

Info

Publication number: CN115237724A
Application number: CN202210927805.5A
Authority: CN
Inventors: 葛春健
Original assignee: Ping An Property and Casualty Insurance Company of China Ltd
Current assignee: Ping An Property and Casualty Insurance Company of China Ltd
Priority date: 2022-08-03
Filing date: 2022-08-03
Publication date: 2022-10-25

Abstract

The embodiment of the application belongs to the field of artificial intelligence and relates to a data monitoring method based on artificial intelligence, which comprises the following steps: acquiring processing state data of each service node based on the data reporting component; performing polymerization processing on the processing state data based on a data engine to obtain target index data; calling a preset number of abnormality analysis models, and determining a target abnormality analysis model from all abnormality analysis models; performing anomaly prediction processing on the target index data based on the target anomaly analysis model to generate a corresponding anomaly prediction result; and carrying out identification processing on the target index data based on the abnormal prediction result and displaying the target index data. The application also provides a data monitoring device, computer equipment and a storage medium based on artificial intelligence. In addition, the application also relates to a block chain technology, and the target index data can be stored in the block chain. The method and the device improve the processing efficiency and accuracy of the abnormity analysis of the target index data.

Description

Data monitoring method, device, equipment and storage medium based on artificial intelligence

Technical Field

The present application relates to the field of artificial intelligence technologies, and in particular, to a method, an apparatus, a device, and a storage medium for data monitoring based on artificial intelligence.

Background

In the operation process of the service system, it is usually necessary to collect data of the buried point related to the user, and further monitor and process the collected data of the buried point, and sometimes find an abnormal situation existing in the service system. In the existing monitoring mode for data of buried points, operation and maintenance personnel usually perform manual anomaly analysis on the data of the buried points according to some predefined monitoring rules and wind control rules, and then need to arrange the obtained analysis results. Such a data monitoring and processing method needs to consume more human resources, has low processing efficiency, cannot ensure the accuracy of anomaly analysis, and is difficult to meet the increasing high-quality requirement of service system monitoring.

Disclosure of Invention

An embodiment of the present application aims to provide a data monitoring method and apparatus, a computer device, and a storage medium based on artificial intelligence, so as to solve the technical problems that the existing data monitoring processing method needs to consume more human resources, has low processing efficiency, and cannot ensure the accuracy of anomaly analysis.

In order to solve the above technical problem, an embodiment of the present application provides a data monitoring method based on artificial intelligence, which adopts the following technical solutions:

collecting processing state data of each service node based on a preset data reporting component;

performing aggregation processing on the processing state data based on a preset data engine to obtain target index data corresponding to the processing state data;

calling a preset number of anomaly analysis models, and determining a target anomaly analysis model from all the anomaly analysis models; wherein the number of the target anomaly analysis models is one or more;

performing anomaly prediction processing on the target index data based on the target anomaly analysis model to generate an anomaly prediction result corresponding to the target index data;

and performing identification processing on the target index data based on the abnormal prediction result to obtain processed target index data, and displaying the processed target index data.

Further, the step of performing aggregation processing on the processing state data based on a preset data engine to obtain target index data corresponding to the processing state data specifically includes:

acquiring preset time dimension information and partition conditions;

calling the data engine;

and performing classification aggregation operation on the processing state data according to the time dimension information and the partition conditions by using the data engine to obtain the target index data.

Further, the step of calling a preset number of the target abnormality analysis models and determining the target abnormality analysis model from all the abnormality analysis models includes:

acquiring a preset test sample data set;

generating a recall rate of each anomaly analysis model based on the test sample data set;

generating a prediction accuracy rate of each abnormal analysis model based on the test sample data set;

generating a predicted processing efficiency value of each abnormal analysis model based on the test sample data set;

calling a preset calculation formula to generate a comprehensive processing score of each abnormal analysis model based on the recall rate, the prediction accuracy rate and the prediction processing efficiency value of each abnormal analysis model;

and screening the abnormal analysis model with the maximum comprehensive processing score from all the abnormal analysis models to obtain the target abnormal analysis model.

Further, the step of generating the predicted processing efficiency value of each anomaly analysis model based on the test sample data set specifically includes:

acquiring the test sample data set; wherein the set of test sample data comprises a plurality of test sample data;

when a specified anomaly analysis model acquires each piece of test sample data, respectively counting the prediction processing time of the specified anomaly analysis model for generating a prediction result corresponding to each piece of test sample data; the specified anomaly analysis model is any one of all the anomaly analysis models;

deleting the first prediction processing time with the largest numerical value and the second prediction processing time with the smallest numerical value from all the prediction processing times to obtain third prediction processing time;

calculating a first average value between all the third prediction processing times;

acquiring appointed prediction processing time corresponding to the median of all the prediction processing time;

and calculating a second average value between the first average value and the specified prediction processing time, and taking the second average value as the prediction processing efficiency value of the specified anomaly analysis model.

Further, the step of performing an anomaly prediction process on the target index data based on the target anomaly analysis model to generate an anomaly prediction result corresponding to the target index data includes:

performing anomaly prediction processing on the target index data by using each target anomaly analysis model to obtain a plurality of corresponding specified anomaly prediction results; wherein the specified anomaly prediction result comprises data normality or data anomaly;

judging whether all the specified abnormal prediction results are normal data;

if all the specified abnormal prediction results are data normal, generating a first abnormal prediction result with normal data corresponding to the target index data;

and if at least one abnormal prediction result with data abnormality exists in all the specified abnormal prediction results, generating a second abnormal prediction result with data abnormality corresponding to the target index data.

Further, before the step of calling the preset number of anomaly analysis models, the method further includes:

acquiring a preset number of training sample data sets, and determining a specified training sample data set from the training sample data sets; the appointed training sample data set is any one data set in all the training sample data sets, and comprises a plurality of appointed index data samples and appointed class labels corresponding to the appointed index data samples;

training a preset machine learning model based on the specified index data sample and the specified class label to obtain an original anomaly analysis model;

acquiring a preset verification sample data set, and verifying the original anomaly analysis model based on the verification sample data set;

and if the original anomaly analysis model passes the verification, taking the original anomaly analysis model as a specified anomaly analysis model corresponding to the specified training sample data set.

Further, after the step of performing an anomaly prediction process on the target index data based on the target anomaly analysis model to generate an anomaly prediction result corresponding to the target index data, the method further includes:

judging whether the abnormal prediction result is data abnormal or not;

if the data is abnormal, generating alarm information corresponding to the target index data based on the abnormal prediction result and a preset alarm information template;

acquiring a communication address of a target user;

and sending the alarm information to the communication address.

In order to solve the above technical problem, an embodiment of the present application further provides a data monitoring apparatus based on artificial intelligence, which adopts the following technical solutions:

the acquisition module is used for acquiring processing state data of each service node based on a preset data reporting component;

the first generation module is used for carrying out aggregation processing on the processing state data based on a preset data engine to obtain target index data corresponding to the processing state data;

the determining module is used for calling a preset number of abnormal analysis models and determining a target abnormal analysis model from all the abnormal analysis models; wherein the number of the target anomaly analysis models is one or more;

the second generation module is used for carrying out abnormity prediction processing on the target index data based on the target abnormity analysis model and generating an abnormity prediction result corresponding to the target index data;

and the display module is used for identifying the target index data based on the abnormal prediction result to obtain processed target index data and displaying the processed target index data.

In order to solve the above technical problem, an embodiment of the present application further provides a computer device, which adopts the following technical solutions:

calling a preset number of abnormality analysis models, and determining a target abnormality analysis model from all the abnormality analysis models; wherein the number of the target anomaly analysis models is one or more;

In order to solve the foregoing technical problem, an embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solutions:

Compared with the prior art, the embodiment of the application mainly has the following beneficial effects:

according to the embodiment of the application, after the processing state data of each service node is acquired based on the preset data reporting component, the processing state data are subjected to aggregation processing based on the preset data engine to obtain the target index data corresponding to the processing state data, then the preset number of abnormal analysis models are called, the target abnormal analysis models are determined from all the abnormal analysis models, then the target abnormal analysis models are subjected to abnormal prediction processing based on the target abnormal analysis models to generate the abnormal prediction results corresponding to the target index data, finally the target index data are subjected to identification processing based on the abnormal prediction results to obtain the processed target index data, and the processed target index data are displayed. According to the method and the device, processing state data generated by the user at each service node can be processed based on the computing engine to generate corresponding index data, and then the preset anomaly analysis model can be used for carrying out anomaly prediction processing on the target index data, so that a corresponding anomaly prediction result can be generated quickly and accurately, and the processing efficiency and accuracy of anomaly analysis on the target index data are improved. In addition, the target index data are identified and displayed based on the obtained abnormal prediction result, so that the intelligence and convenience of target index data display are improved, and the use experience of a user is improved.

Drawings

In order to more clearly illustrate the solution of the present application, the drawings needed for describing the embodiments of the present application will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained by those skilled in the art without inventive effort.

FIG. 1 is an exemplary system architecture diagram in which the present application may be applied;

FIG. 2 is a flow diagram of one embodiment of an artificial intelligence based data monitoring method according to the present application;

FIG. 3 is a schematic block diagram of one embodiment of an artificial intelligence based data monitoring apparatus according to the present application;

FIG. 4 is a schematic block diagram of one embodiment of a computer device according to the present application.

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "including" and "having," and any variations thereof, in the description and claims of this application and the description of the above figures are intended to cover non-exclusive inclusions. The terms "first," "second," and the like in the description and claims of this application or in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. Network 104 is the medium used to provide communication links between

terminal devices

101, 102, 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have various communication client applications installed thereon, such as a web browser application, a shopping application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like.

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to a smart phone, a tablet computer, an e-book reader, an MP3 player (Moving Picture experts Group Audio Layer III, motion Picture experts compression standard Audio Layer 3), an MP4 player (Moving Picture experts Group Audio Layer IV, motion Picture experts compression standard Audio Layer 4), a laptop portable computer, a desktop computer, and the like.

The server 105 may be a server providing various services, such as a background server providing support for pages displayed on the

terminal devices

101, 102, 103.

It should be noted that the artificial intelligence based data monitoring method provided in the embodiments of the present application is generally executed by a server/terminal device, and accordingly, an artificial intelligence based data monitoring apparatus is generally disposed in the server/terminal device.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for an implementation.

With continued reference to FIG. 2, a flow diagram of one embodiment of an artificial intelligence based data monitoring method according to the present application is shown. The data monitoring method based on artificial intelligence comprises the following steps:

step S201, collecting processing status data of each service node based on a preset data reporting component.

In this embodiment, the electronic device (for example, the server/terminal device shown in fig. 1) on which the artificial intelligence-based data monitoring method operates may collect the processing state data of each service node in a wired connection manner or a wireless connection manner. It should be noted that the wireless connection means may include, but is not limited to, a 3G/4G/5G connection, a WiFi connection, a bluetooth connection, a WiMAX connection, a Zigbee connection, an UWB (ultra wideband) connection, and other wireless connection means now known or developed in the future. The data reporting component is an instrumentation reporting component introduced by developers according to actual service requirements, and the data reporting component reports the data of the event node based on a code layer. The service node may be an insurance application service node, and the processing state data of the service node may include event state data acquired during the stages of insurance browsing, insurance quotation, insurance application, payment, underwriting and the like of the user. In addition, after the processing state data of each service node is obtained, the processing state data can be uniformly reported to the data engine according to the sequence of events. In addition, business personnel can define the relationship between nodes and links reported by events in the reporting component according to an actual business processing flow (such as a insurance application business flow), so that when target index data are displayed subsequently, the business personnel can quickly know the conversion index data of each link in the insurance application business processing flow, can find business problems in time, and can quickly locate reasons aiming at the customer complaint problems, thereby improving response timeliness, being beneficial to helping insurance enterprises to optimize the insurance application flow links and improving the satisfaction degree of user insurance purchase.

Step S202, carrying out aggregation processing on the processing state data based on a preset data engine to obtain target index data corresponding to the processing state data.

In this embodiment, the data engine may be specifically Flink. Flink is an open source stream processing framework developed by the Apache software Foundation, with the core of a distributed stream data engine written in Java and Scale. Flink executes arbitrary streaming data programs in a data parallel (distributed) and pipelined manner, and Flink's pipelined runtime system can execute batch and stream processing programs. The specific implementation process of performing aggregation processing on the processing state data based on the preset data engine to obtain the target index data corresponding to the processing state data is described in further detail in the following specific embodiments, and is not set forth herein.

Step S203, calling a preset number of abnormal analysis models, and determining a target abnormal analysis model from all the abnormal analysis models; wherein the number of the target abnormality analysis models is one or more.

In this embodiment, the value of the preset number is not specifically limited, and may be set according to actual use requirements. In addition, the above specific implementation process of determining the target anomaly analysis model from all the anomaly analysis models is further described in detail in the following specific embodiments, and is not set forth herein in any way. Further, the determination process for the number of target abnormality analysis models may include: and acquiring available memories of the electronic equipment, and inquiring the number corresponding to the available memories based on a preset available memory-number mapping table to be used as the number of the standard anomaly analysis models. The available memory-quantity mapping table may be generated based on test results, on-line problem analysis, and expert experience, and is a data table in which mapping relationships between available memory information and model operation quantities are recorded. By inquiring the corresponding model number from the available memory-number mapping table based on the current available memory information of the electronic equipment, the loss cost of the data calculation processing process of the internal model of the electronic equipment can be ensured to be low, and the data calculation processing is ensured to be at a normal speed and the electronic equipment cannot be blocked.

Step S204, carrying out abnormity prediction processing on the target index data based on the target abnormity analysis model, and generating an abnormity prediction result corresponding to the target index data.

In this embodiment, the specific implementation process of performing the anomaly prediction processing on the target index data based on the target anomaly analysis model to generate the anomaly prediction result corresponding to the target index data is described in further detail in the following specific embodiments, and will not be described in detail here.

Step S205, performing identification processing on the target index data based on the abnormal prediction result to obtain processed target index data, and displaying the processed target index data.

In this embodiment, the content of the anomaly prediction result may include data normal or data anomaly. The process of identifying the target index data based on the abnormal prediction result may include: if the abnormal prediction result is data normality, green labeling can be carried out on the target index data to identify the target index data as normal data through green labeling, and if the abnormal prediction result is data abnormality, red labeling can be carried out on the target index data to identify the target index data as abnormal data through red labeling. By marking the target index data in a normal state or an abnormal state, the user can clearly check the data state of the target index data, and the use experience of the user is improved. In addition, the obtained processed target index data can be displayed through a preset visual query interface.

According to the method and the device, after processing state data of each service node are acquired based on a preset data reporting component, the processing state data are subjected to aggregation processing based on a preset data engine to obtain target index data corresponding to the processing state data, then a preset number of abnormity analysis models are called, a target abnormity analysis model is determined from all abnormity analysis models, abnormity prediction processing is carried out on the target index data based on the target abnormity analysis model subsequently, an abnormity prediction result corresponding to the target index data is generated, finally, identification processing is carried out on the target index data based on the abnormity prediction result to obtain the processed target index data, and the processed target index data are displayed. According to the method and the device, processing state data generated by the user at each service node can be processed based on the computing engine to generate corresponding index data, and then the preset anomaly analysis model can be used for carrying out anomaly prediction processing on the target index data, so that a corresponding anomaly prediction result can be generated quickly and accurately, and the processing efficiency and accuracy of anomaly analysis on the target index data are improved. In addition, the target index data are identified and displayed based on the obtained abnormal prediction result, so that the intelligence and convenience of target index data display are improved, and the use experience of a user is improved.

In some optional implementations, step S202 includes the following steps:

and acquiring preset time dimension information and partition conditions.

In this embodiment, the preset time dimension information and the partition condition may be set according to actual service usage requirements. For example, the dimension information may include any one of time dimensions in terms of minutes/hours/days.

And calling the data engine.

In this embodiment, the data engine may be a Flink.

And performing classification and aggregation operation on the processing state data according to the time dimension information and the partition conditions by using the data engine to obtain the target index data.

In this embodiment, the Flink may perform a sort and group operation according to a time window according to the time dimension information and the partition condition, so as to obtain target index data corresponding to the processing status data. In addition, the generated target index data may be stored, for example, the target index data may be written into an Elasticsearch server and stored.

After the processing state data of each service node is collected, the processing state data can be operated by using the calculation engine, so that the required target index data can be generated quickly, and the generation rate and accuracy of the index data are ensured.

In some optional implementation manners of this embodiment, the number of the target abnormality analysis models is one, and step S203 includes the following steps:

and acquiring a preset test sample data set.

In this embodiment, the test sample data set may be generated based on a training sample data set acquired in advance, for example, data of a preset proportion may be randomly acquired from the training sample data set as the test sample data set. The value of the preset ratio is not particularly limited, and may be set according to actual requirements, for example, may be set to 30%. In addition, the test sample set includes a plurality of test sample data and a category label corresponding to each test sample data.

And generating the recall rate of each abnormal analysis model based on the test sample data set.

In this embodiment, the recall rate of generating the anomaly analysis model can be calculated by an error matrix (or called a confusion matrix). The confusion matrix is used to measure the accuracy of a classifier. For the dichotomy problem, the samples are divided into four cases, namely True Positive (TP), false Positive (FP), true Negative (TN) and False Negative (FN) according to the combination of the True classes and the prediction classes of the classifier. Based on the error matrix, the recall rate of the anomaly analysis model can be calculated using the following calculation formula: recall = TP/(TP + FN).

And generating the prediction accuracy rate of each anomaly analysis model based on the test sample data set.

In this embodiment, based on the error matrix, the following calculation formula can be used to calculate the prediction accuracy of the anomaly analysis model: prediction accuracy = TP/(TP + FP).

And generating the predicted processing efficiency value of each abnormal analysis model based on the test sample data set.

In this embodiment, the above-mentioned specific implementation process of generating the predicted processing efficiency value of each anomaly analysis model based on the test sample data set is further described in detail in the following specific embodiments, and is not described in detail herein.

And calling a preset calculation formula to generate a comprehensive processing score of each abnormal analysis model based on the recall rate, the prediction accuracy rate and the prediction processing efficiency value of each abnormal analysis model.

In this embodiment, the predetermined calculation formula is Score = x a + y b + z c, where Score is the integrated processing Score, x is the recall ratio, a is a first weight of the recall ratio, y is the prediction precision ratio, b is a second weight of the prediction precision ratio, z is the predicted processing efficiency value, and c is a third weight of the predicted processing efficiency value. In addition, the values of the first weight, the second weight, and the third weight are not particularly limited, and may be set according to actual service requirements, and preferably, the second weight > the first weight > the third weight, and a sum of the first weight, the second weight, and the third weight is equal to 1.

And screening the abnormal analysis model with the largest comprehensive processing score from all the abnormal analysis models to obtain the target abnormal analysis model.

According to the method and the device, the comprehensive processing score of each abnormal analysis model is generated by using the preset test sample set, and then the target abnormal analysis model can be determined based on the obtained comprehensive evaluation score.

In some optional implementations, the generating the predicted processing efficiency value of each anomaly analysis model based on the test sample data set includes the following steps:

acquiring the test sample data set; wherein the set of test sample data comprises a plurality of test sample data.

When a specified anomaly analysis model obtains each test sample data, respectively counting the prediction processing time of the specified anomaly analysis model for generating a prediction result corresponding to each test sample data; wherein the specified anomaly analysis model is any one of all the anomaly analysis models.

In this embodiment, for example, if the time when the specified anomaly analysis model successfully receives a piece of test sample data is t1, and the time when the specified anomaly analysis model successfully generates the prediction result corresponding to the test sample data is t2, it is determined that the predicted processing time t of the specified anomaly analysis model corresponding to the test sample data is equal to t2-t1.

And deleting the first prediction processing time with the largest numerical value and the second prediction processing time with the smallest numerical value from all the prediction processing times to obtain third prediction processing time.

Calculating a first average value between all the third prediction processing times.

In this embodiment, the sum of all the third prediction processing times may be calculated first, then the number of all the third prediction processing times is obtained, and then the quotient between the sum and the number is calculated, where the obtained quotient is the first average value.

And acquiring the appointed prediction processing time corresponding to the median of all the prediction processing times.

In this embodiment, the process of calculating the second average value may refer to the process of calculating the first average value, which is not described herein too much.

The method and the device can rapidly calculate the predicted processing efficiency value of each abnormal analysis model by using the test sample data set, are favorable for accurately screening all abnormal analysis models based on the predicted processing efficiency value to determine the target abnormal analysis model with the highest comprehensive processing capacity, and further can use the obtained target abnormal analysis model to perform abnormal analysis on target index data, so that the accuracy of the obtained abnormal prediction result corresponding to the target index data is effectively improved, and the processing efficiency of the abnormal prediction of the target index data is improved.

In some optional implementations, the number of the target anomaly analysis models is multiple, and step S204 includes the following steps:

performing anomaly prediction processing on the target index data by using each target anomaly analysis model to obtain a plurality of corresponding specified anomaly prediction results; wherein the specified anomaly prediction result comprises data normality or data anomaly.

And judging whether all the specified abnormal prediction results are normal data.

And if all the specified abnormal prediction results are normal data, generating a first abnormal prediction result which is normal data and corresponds to the target index data.

According to the method and the device, the obtained target index data are subjected to the abnormal prediction processing by using the plurality of target abnormal analysis models, the final abnormal prediction result for the target index data is generated according to the prediction results generated by the target abnormal analysis models, the comprehensiveness and the accuracy of the generated abnormal prediction result are effectively ensured, the phenomenon that the prediction error is overlarge due to the fact that only one abnormal analysis model is used for performing the prediction processing on the target index data is avoided, the prediction accuracy of the target index data is effectively improved, and the generation intelligence of the abnormal prediction result is improved.

In some optional implementation manners of this embodiment, after step S203, the electronic device may further perform the following steps:

acquiring a preset number of training sample data sets, and determining a specified training sample data set from the training sample data sets; the appointed training sample data set is any one data set in all the training sample data sets, and comprises a plurality of appointed index data samples and appointed category labels corresponding to the appointed index data samples.

In this embodiment, the tag content of the above-mentioned tag of the specified category may include data normal or data abnormal.

And training a preset machine learning model based on the specified index data sample and the specified class label to obtain an original anomaly analysis model.

In this embodiment, the machine learning model may include any one of a random forest model, a naive bayes model, and a logistic regression model. In addition, the training process for various machine learning models can refer to the existing model training process, and is not elaborated herein.

And acquiring a preset verification sample data set, and verifying the original anomaly analysis model based on the verification sample data set.

In this embodiment, the verification of the original anomaly analysis model based on the verification sample data set means whether a prediction accuracy obtained by performing anomaly prediction processing on the verification sample data set by the statistical original anomaly analysis model is greater than a preset accuracy threshold, and if the prediction accuracy is greater than the accuracy threshold, it is determined that the original anomaly analysis model is converged, so as to end the training of the original anomaly analysis model, and the original anomaly analysis model after the training is ended is used as the specified anomaly analysis model. In addition, if the prediction accuracy is smaller than the accuracy threshold, it indicates that the training of the trained original anomaly analysis model has not reached the preset standard, and may be that the number of samples of the training sample data set used for training is too small or the number of samples of the verification sample data set is too small, so in this case, the number of samples of the specified index data sample may be further increased, for example, a fixed number is further increased each time or a random number is increased each time, then the above-mentioned training process and testing process are executed again on this basis, and the above-mentioned steps are executed in a loop until the prediction accuracy of the trained original anomaly analysis model meets the requirement that the prediction accuracy is larger than the accuracy threshold, and the model training is ended.

The preset machine learning model is trained and verified by using the training sample data set comprising the designated index data sample and the designated class label corresponding to the index data sample, so that the abnormal analysis model meeting the actual use requirement can be intelligently and quickly generated, the abnormal prediction of the target index data can be performed on the basis of the abnormal analysis model obtained by training when the target index data is generated, the abnormal prediction result corresponding to the target index data can be quickly and accurately generated, and the intelligence and the efficiency of generating the abnormal prediction result are improved.

In some optional implementation manners of this embodiment, after step S204, the electronic device may further perform the following steps:

and judging whether the abnormal prediction result is data abnormality or not.

In this embodiment, the content of the anomaly prediction result includes data normality or data anomaly.

And if the data is abnormal, generating alarm information corresponding to the target index data based on the abnormal prediction result and a preset alarm information template.

In this embodiment, the abnormal prediction result of the target index data may be input into a pre-created alarm information template, so that alarm information corresponding to the target index data may be generated. The alarm information template can be compiled and generated according to actual use requirements.

And acquiring the communication address of the target user.

In this embodiment, the target user may be a manager related to data monitoring operation. In addition, the communication address can be a mail address.

And sending the alarm information to the communication address.

In this embodiment, if the communication address is a mail address, the warning message may be sent to the communication address of the target user by logging in to the mail server and based on the mail server.

According to the method and the device, after the abnormal prediction result of the data abnormality corresponding to the target index data is generated through the target abnormal analysis model, the alarm information corresponding to the target index data can be generated intelligently based on the preset alarm information template, and the alarm information is sent to the communication address of the relevant target user, so that the target user can timely perform corresponding processing on the target index data according to the received alarm information, the processing efficiency of the abnormal data is improved, and the use experience of the target user is also improved.

It is emphasized that, to further ensure the privacy and security of the target index data, the target index data may also be stored in a node of a block chain.

The block chain referred by the application is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware associated with computer readable instructions, which can be stored in a computer readable storage medium, and when executed, the processes of the embodiments of the methods described above can be included. The storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless otherwise indicated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

With further reference to fig. 3, as an implementation of the method shown in fig. 2, the present application provides an embodiment of an artificial intelligence based data monitoring apparatus, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.

As shown in fig. 3, the artificial intelligence based data monitoring apparatus 300 according to the present embodiment includes: an acquisition module 301, a first generation module 302, a determination module 303, a second generation module 304, and a presentation module 305. Wherein:

an acquisition module 301, configured to acquire processing state data of each service node based on a preset data reporting component;

a first generation module 302, configured to perform aggregation processing on the processing state data based on a preset data engine to obtain target index data corresponding to the processing state data;

the determining module 303 is configured to call a preset number of anomaly analysis models, and determine a target anomaly analysis model from all the anomaly analysis models; wherein the number of the target anomaly analysis models is one or more;

a second generating module 304, configured to perform an anomaly prediction process on the target index data based on the target anomaly analysis model, and generate an anomaly prediction result corresponding to the target index data;

a display module 305, configured to perform identification processing on the target index data based on the abnormal prediction result to obtain processed target index data, and display the processed target index data.

In this embodiment, the operations respectively executed by the modules or units correspond to the steps of the artificial intelligence based data monitoring method of the foregoing embodiment one to one, and are not described herein again.

In some optional implementations of the present embodiment, the first generating module 302 includes:

the first acquisition submodule is used for acquiring preset time dimension information and partition conditions;

the calling submodule is used for calling the data engine;

and the operation submodule is used for performing classification and aggregation operation on the processing state data according to the time dimension information and the partition conditions by using the data engine to obtain the target index data.

In this embodiment, the operations respectively executed by the modules or units correspond to the steps of the artificial intelligence based data monitoring method of the foregoing embodiment one by one, and are not described herein again.

In some optional implementations of this embodiment, the number of the target abnormality analysis models is one, and the determining module 303 includes:

the second obtaining submodule is used for obtaining a preset test sample data set;

the first generation submodule is used for generating the recall rate of each anomaly analysis model based on the test sample data set;

the second generation submodule is used for generating the prediction accuracy rate of each abnormal analysis model based on the test sample data set;

a third generation submodule, configured to generate a predicted processing efficiency value of each anomaly analysis model based on the test sample data set;

the fourth generation submodule is used for calling a preset calculation formula to generate a comprehensive processing score of each abnormal analysis model based on the recall rate, the prediction accuracy rate and the prediction processing efficiency value of each abnormal analysis model;

and the determining submodule is used for screening the abnormal analysis model with the largest comprehensive processing score from all the abnormal analysis models to obtain the target abnormal analysis model.

In this embodiment, the operations that the modules or units are respectively configured to execute correspond to the steps of the artificial intelligence based data monitoring method in the foregoing embodiment one to one, and are not described herein again.

In some optional implementations of this embodiment, the third generating sub-module includes:

a first obtaining unit, configured to obtain the test sample data set; wherein the set of test sample data comprises a plurality of test sample data;

the statistical unit is used for respectively counting the prediction processing time of the specified anomaly analysis model for generating the prediction result corresponding to each test sample data when the specified anomaly analysis model obtains each test sample data; the specified anomaly analysis model is any one of all the anomaly analysis models;

a deleting unit, configured to delete a first prediction processing time with a largest numerical value and a second prediction processing time with a smallest numerical value from all the prediction processing times, so as to obtain a third prediction processing time;

a calculation unit configured to calculate a first average value between all the third prediction processing times;

a second acquisition unit configured to acquire a specified prediction processing time corresponding to a median of all the prediction processing times;

and the determining unit is used for calculating a second average value between the first average value and the specified prediction processing time, and taking the second average value as the predicted processing efficiency value of the specified abnormity analysis model.

In some optional implementations of this embodiment, the number of the target anomaly analysis models is multiple, and the second generating module 304 includes:

the prediction submodule is used for performing anomaly prediction processing on the target index data by using each target anomaly analysis model to obtain a plurality of corresponding specified anomaly prediction results; wherein the specified anomaly prediction result comprises data normality or data anomaly;

the judging submodule is used for judging whether all the specified abnormal prediction results are normal data or not;

a fifth generation submodule, configured to generate a first abnormal prediction result with normal data, where the first abnormal prediction result is corresponding to the target index data, if all the specified abnormal prediction results are data normal;

and a sixth generation submodule, configured to generate a second abnormal prediction result of the data abnormality corresponding to the target index data if at least one abnormal prediction result of the data abnormality exists in all the specified abnormal prediction results.

In some optional implementations of this embodiment, the artificial intelligence based data monitoring apparatus further includes:

the device comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring training sample data sets with a preset number and determining a specified training sample data set from the training sample data sets; the appointed training sample data set is any one data set in all the training sample data sets, and comprises a plurality of appointed index data samples and appointed class labels corresponding to the appointed index data samples;

the training module is used for training a preset machine learning model based on the specified index data sample and the specified class label to obtain an original anomaly analysis model;

the verification module is used for acquiring a preset verification sample data set and verifying the original anomaly analysis model based on the verification sample data set;

and the third generation module is used for taking the original abnormal analysis model as a specified abnormal analysis model corresponding to the specified training sample data set if the original abnormal analysis model passes the verification.

the judging module is used for judging whether the abnormal prediction result is data abnormality;

a fourth generating module, configured to generate alarm information corresponding to the target index data based on the abnormal prediction result and a preset alarm information template if the data is abnormal;

the second acquisition module is used for acquiring the communication address of the target user;

and the sending module is used for sending the alarm information to the communication address.

In order to solve the technical problem, an embodiment of the present application further provides a computer device. Referring to fig. 4, fig. 4 is a block diagram of a basic structure of a computer device according to the present embodiment.

The computer device 4 comprises a memory 41, a processor 42, a network interface 43 communicatively connected to each other via a system bus. It is noted that only computer device 4 having components 41-43 is shown, but it is understood that not all of the shown components are required to be implemented, and that more or fewer components may be implemented instead. As will be understood by those skilled in the art, the computer device is a device capable of automatically performing numerical calculation and/or information processing according to a preset or stored instruction, and the hardware includes, but is not limited to, a microprocessor, an Application Specific Integrated Circuit (ASIC), a Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), an embedded device, and the like.

The computer device can be a desktop computer, a notebook, a palm computer, a cloud server and other computing devices. The computer equipment can carry out man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch panel or voice control equipment and the like.

The memory 41 includes at least one type of readable storage medium including flash memory, hard disks, multimedia cards, card-type memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disks, optical disks, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as a hard disk or a memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the computer device 4. Of course, the memory 41 may also include both internal and external storage devices of the computer device 4. In this embodiment, the memory 41 is generally used for storing an operating system and various application software installed in the computer device 4, such as computer readable instructions of an artificial intelligence based data monitoring method. Further, the memory 41 may also be used to temporarily store various types of data that have been output or are to be output.

The processor 42 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is configured to execute computer readable instructions stored in the memory 41 or process data, for example, execute computer readable instructions of the artificial intelligence based data monitoring method.

The network interface 43 may comprise a wireless network interface or a wired network interface, and the network interface 43 is generally used for establishing a communication connection between the computer device 4 and other electronic devices.

in the embodiment of the application, after the processing state data of each service node is acquired based on a preset data reporting component, the processing state data is subjected to aggregation processing based on a preset data engine to obtain target index data corresponding to the processing state data, then a preset number of abnormity analysis models are called, a target abnormity analysis model is determined from all abnormity analysis models, abnormity prediction processing is subsequently performed on the target index data based on the target abnormity analysis model to generate an abnormity prediction result corresponding to the target index data, and finally the target index data is subjected to identification processing based on the abnormity prediction result to obtain the processed target index data, and the processed target index data is displayed. According to the method and the device, processing state data generated by the user at each service node can be processed based on the calculation engine to generate corresponding index data, and then a preset anomaly analysis model can be used for carrying out anomaly prediction processing on the target index data, so that a corresponding anomaly prediction result can be rapidly and accurately generated, and the processing efficiency and accuracy of anomaly analysis on the target index data are improved. In addition, the target index data are identified and displayed based on the obtained abnormal prediction result, so that the intelligence and convenience of target index data display are improved, and the use experience of a user is improved.

The present application further provides another embodiment, which is to provide a computer-readable storage medium storing computer-readable instructions executable by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence based data monitoring method as described above.

in the embodiment of the application, after the processing state data of each service node is acquired based on a preset data reporting component, the processing state data is subjected to aggregation processing based on a preset data engine to obtain target index data corresponding to the processing state data, then a preset number of abnormal analysis models are called, a target abnormal analysis model is determined from all abnormal analysis models, then abnormal prediction processing is performed on the target index data based on the target abnormal analysis model, an abnormal prediction result corresponding to the target index data is generated, finally the target index data is subjected to identification processing based on the abnormal prediction result to obtain the processed target index data, and the processed target index data is displayed. According to the method and the device, processing state data generated by the user at each service node can be processed based on the calculation engine to generate corresponding index data, and then a preset anomaly analysis model can be used for carrying out anomaly prediction processing on the target index data, so that a corresponding anomaly prediction result can be rapidly and accurately generated, and the processing efficiency and accuracy of anomaly analysis on the target index data are improved. In addition, the target index data are identified and displayed based on the obtained abnormal prediction result, so that the intelligence and convenience of target index data display are improved, and the use experience of a user is improved.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.

It is to be understood that the above-described embodiments are merely illustrative of some, but not restrictive, of the broad invention, and that the appended drawings illustrate preferred embodiments of the invention and do not limit the scope of the invention. This application is capable of embodiments in many different forms and the embodiments are provided so that this disclosure will be thorough and complete. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to one skilled in the art that the present application may be practiced without modification or with equivalents of some of the features described in the foregoing embodiments. All equivalent structures made by using the contents of the specification and the drawings of the present application are directly or indirectly applied to other related technical fields, and all the equivalent structures are within the protection scope of the present application.

Claims

1. A data monitoring method based on artificial intelligence is characterized by comprising the following steps:

2. The artificial intelligence-based data monitoring method according to claim 1, wherein the step of performing aggregation processing on the processing state data based on a preset data engine to obtain target index data corresponding to the processing state data specifically includes:

acquiring preset time dimension information and partition conditions;

calling the data engine;

3. The artificial intelligence-based data monitoring method according to claim 1, wherein the number of the target anomaly analysis models is one, and the step of calling a preset number of the anomaly analysis models and determining the target anomaly analysis model from all the anomaly analysis models specifically includes:

acquiring a preset test sample data set;

4. The artificial intelligence based data monitoring method of claim 3, wherein the step of generating the predicted processing efficiency value of each of the anomaly analysis models based on the test sample data set specifically comprises:

when a specified anomaly analysis model obtains each test sample data, respectively counting the prediction processing time of the specified anomaly analysis model for generating a prediction result corresponding to each test sample data; the specified anomaly analysis model is any one of all the anomaly analysis models;

and calculating a second average value between the first average value and the specified prediction processing time, and taking the second average value as the prediction processing efficiency value of the specified abnormity analysis model.

5. The artificial intelligence-based data monitoring method according to claim 1, wherein the number of the target anomaly analysis models is plural, and the step of performing anomaly prediction processing on the target index data based on the target anomaly analysis models to generate an anomaly prediction result corresponding to the target index data specifically includes:

judging whether all the specified abnormal prediction results are normal data;

6. The artificial intelligence based data monitoring method of claim 1, further comprising, prior to the step of invoking a preset number of anomaly analysis models:

7. The artificial intelligence based data monitoring method according to claim 1, wherein after the step of performing an anomaly prediction process on the target index data based on the target anomaly analysis model to generate an anomaly prediction result corresponding to the target index data, the method further comprises:

judging whether the abnormal prediction result is data abnormal or not;

acquiring a communication address of a target user;

and sending the alarm information to the communication address.

8. A data monitoring device based on artificial intelligence, comprising:

9. A computer device comprising a memory having computer readable instructions stored therein and a processor that when executed performs the steps of the artificial intelligence based data monitoring method of any one of claims 1 to 7.

10. A computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, implement the steps of the artificial intelligence based data monitoring method of any one of claims 1 to 7.