LU501931B1 - Data exception analysis method and device - Google Patents

Data exception analysis method and device Download PDF

Info

Publication number
LU501931B1
LU501931B1 LU501931A LU501931A LU501931B1 LU 501931 B1 LU501931 B1 LU 501931B1 LU 501931 A LU501931 A LU 501931A LU 501931 A LU501931 A LU 501931A LU 501931 B1 LU501931 B1 LU 501931B1
Authority
LU
Luxembourg
Prior art keywords
data
exception
variable
exceptional
cluster
Prior art date
Application number
LU501931A
Other languages
German (de)
Inventor
Lele Wang
Original Assignee
Univ Zhengzhou Aeronautics
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Univ Zhengzhou Aeronautics filed Critical Univ Zhengzhou Aeronautics
Priority to LU501931A priority Critical patent/LU501931B1/en
Application granted granted Critical
Publication of LU501931B1 publication Critical patent/LU501931B1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/11—File system administration, e.g. details of archiving or snapshots
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0751—Error or fault detection not based on redundancy
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention includes: obtaining service components, servers, process information and reference indicators of a target data cluster; obtaining monitoring indicators and exceptional variables corresponding to the service components, servers, process information and reference indicators of the target data cluster according to a preset correspondence; performing data collection on the target data cluster according to the monitoring indicators and the exceptional variables to obtain to-be-analyzed data; grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain exception prediction results, and storing the exception prediction results. The data exception analysis method of the invention can effectively perform exception prediction on a data cluster. Moreover, exception analysis is performed according to grouping, so as to perform exception prediction on the data cluster from multiple dimensions.

Description

DESCRIPTION LU501931
DATA EXCEPTION ANALYSIS METHOD AND DEVICE Technical Field The present invention relates to a data exception analysis method and device. Background At present, the application of data processing based on big data is very extensive. There are many data clusters in a data platform corresponding to the big data. The data clusters can use different components according to business needs to build a big data platform that adapts to localization. Accordingly, it is crucial to carry out data monitoring for each data cluster in the data platform. At present, a monitoring method for each data cluster in the data platform is to analyze the data cluster that already has an exception. This exception analysis method can only analyze the data cluster already having an exception, but cannot effectively predict the exception of the data cluster, resulting in a poor monitoring effect. Summary of the Invention The present invention provides a data exception analysis method and device to solve a technical problem that the monitoring effect of the existing data exception analysis method is poor.
Disclosed is a data exception analysis method, including: obtaining service components, servers, process information and reference indicators of a target data cluster; obtaining monitoring indicators and exceptional variables corresponding to the service components, servers, process information and reference indicators of the target data cluster according to a preset correspondence; performing data collection on the target data cluster according to the monitoring indicators and the exceptional variables to obtain to-be-analyzed data; grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain exception prediction results; and storing the exception prediction results. LU501931 Further, prior to the step of obtaining the service components, servers, process information and reference indicators of the target data cluster, the data exception analysis method further includes: obtaining cluster IDs of all data clusters stored in a data platform; and according to a preset sorting rule and the cluster IDs of the data clusters, sorting the data clusters to obtain a sorting result, so that data exception analysis is performed on the data clusters according to the sorting result.
Further, the step of grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain exception prediction results specifically includes: selecting the exceptional variables in the to-be-analyzed data according to a preset selection rule and forming a variable group of the exceptional variables selected in a same batch; according to the variable group, respectively obtaining data corresponding to each exceptional variable in the to-be-analyzed data to obtain variable data; and performing exception detection on the variable data respectively to obtain corresponding exception prediction results.
Further, the step of according to the variable group, respectively obtaining data corresponding to each exceptional variable in the to-be-analyzed data to obtain variable data specifically includes: according to each exceptional variable in the variable group, obtaining data carrying the corresponding exceptional variable in the to-be-analyzed data respectively and integrating the data obtained for the same variable group to obtain the variable data.
Further, subsequent to the step of performing exception detection on the variable data respectively to obtain corresponding exception prediction results, the data exception analysis method further includes: if no exceptions occur in all the variable data, obtaining associated data corresponding to the variable data corresponding to the same variable group; and performing exception analysis on the associated data to obtain an additional exception prediction result and adding the additional exception prediction result to the exception prediction results. LU501931 Further, subsequent to the step of grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain the exception prediction results, the data exception analysis method further includes: obtaining exception notification information corresponding to the target data cluster and matching prediction information in the exception prediction results against notification addresses in the exception notification information; and reminding the corresponding notification address of successfully matched prediction information.
Further, the step of obtaining the exception notification information corresponding to the target data cluster and matching the prediction information in the exception prediction results against the notification addresses in the exception notification information specifically includes: obtaining the exception notification information, the exception notification information including at least one notification address, each notification address corresponding to a privilege value: performing exception level evaluation on the prediction information in the exception prediction results to obtain a corresponding exception value; and matching the exception value of the prediction information against the privilege values corresponding to the notification addresses to obtain a privilege value corresponding to the exception value and determining a notification address corresponding to the obtained privilege value.
Further, the step of reminding the corresponding notification address of the successfully matched prediction information specifically includes: obtaining a reminder mode corresponding to the obtained privilege value according to a preset reminder mode database, the reminder mode database including at least two privilege values and reminder modes corresponding to the privilege values; and sending a reminder to the corresponding notification address according to the obtained reminder mode.
Further, the step of storing the exception prediction results specifically includes:
determining a target storage area of the target data cluster according to the cluster ID of tH#/501981 target data cluster and a preset storage relationship, the preset storage relationship including the cluster IDs of all the data clusters stored in the data platform and storage areas corresponding to the cluster IDs of all the data clusters; and storing the exception prediction results in the target storage area.
Disclosed is a data exception analysis device, including a memory, a processor, and a computer program stored on the memory and running on the processor, the processor implementing the foregoing data exception analysis method when executing the computer program.
The technical effect of the data exception analysis method according to the present invention includes: obtaining corresponding monitoring indicators and exceptional variables according to the service components, servers, process information and reference indicators of the target data cluster; then, performing data collection on the target data cluster according to the obtained monitoring indicators and exceptional variables to obtain the to-be-analyzed data; then, grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain the exception prediction results; and finally storing the obtained exception prediction results. The data exception analysis method according to the present invention is a prediction method for exception prediction of data clusters. The method can effectively perform exception prediction on data clusters and improve the monitoring effect and accuracy for the data platform. By grouping the exceptional variables in the to-be-analyzed data and performing exception analysis according to the grouping, exception prediction can be performed on the data clusters from multiple different dimensions based on the multiple variable groups obtained by grouping, thus improving the accuracy of exception prediction for the data clusters. Through data exception prediction, problems can be found before real exceptions occur, so that there can be enough time to solve these problems. Brief Description of Drawings FIG. 11s a flow chart of a data exception analysis method according to the present invention. Detailed Description of Preferred Embodiments Embodiment of a data exception analysis method:
This embodiment provides a data exception analysis method, and the hardware executidi/201931 body of the data exception analysis method may be configured as a computer device, a server device, an intelligent mobile terminal, or the like. Moreover, the application scenario of the hardware execution body of the data exception analysis method is not limited and is set according to actual needs.
As shown in FIG. 1, the data exception analysis method includes the following steps: Step 1: obtaining service components, servers, process information and reference indicators of a target data cluster: In this embodiment, at least one data cluster is stored in a data platform, and the number of data clusters and the specific composition of the data clusters are determined by actual application scenarios. In this embodiment, the data cluster uses at least two or more servers to form a virtual single database logical image to provide data services to a client. Server IDs in communication connection with the data cluster and component IDs corresponding to the data services provided to a user are stored in the local data of the data cluster. The target data cluster belongs to the data clusters in the data platform.
The service components, servers, process information and reference indicators of the target data cluster are obtained. Corresponding service components and servers are queried according to each server ID and component ID in the target data cluster. The process information includes a process running state of each service component and server. The reference indicators refer to indicator parameters preset by the user, and the indicator parameters include a quantity parameter, a running state parameter, or the like.
Usually, there are multiple data clusters in the data platform, so the data clusters need to be sorted to determine a processing order. In this embodiment, prior to step 1, the data exception analysis method further includes: obtaining cluster IDs of all the data clusters stored in the data platform. The cluster IDs may be cluster names of the corresponding data clusters, or may be other unique cluster data information.
According to a preset sorting rule and the cluster IDs of the data clusters, the data clusters are stored to obtain a sorting result. The preset sorting rule is set according to actual needs. For example, if the cluster IDs are the cluster names of the corresponding data cluster, the sorting rule may be the order of initial letters of pinyins of first characters of the cluster names. LU501931 Then, data exception analysis can be performed on each data cluster according to the obtained sorting result, and the higher the sorting of the data cluster, the earlier the data anomaly analysis is performed.
Step 2: obtaining monitoring indicators and exceptional variables corresponding to the service components, servers, process information and reference indicators of the target data cluster according to a preset correspondence: The monitoring indicators and exceptional variables corresponding to the service components, servers, process information and reference indicators of the target data cluster are obtained according to the preset correspondence. In this embodiment, the preset correspondence includes at least two combinations of the service components, the servers, the process information and the reference indicators, and monitoring indicators and exceptional variables corresponding to the combinations. It should be appreciated that, in the preset correspondence, the monitoring indicators and exceptional variables corresponding to different combinations of the service components, the servers, the process information and the reference indicators may be the same or different.
Then, by inputting the service components, servers, process information and reference indicators of the target data cluster into the preset correspondence, the monitoring indicators and exceptional variables corresponding to the service components, servers, process information and reference indicators of the target data cluster can be obtained.
Step 3: performing data collection on the target data cluster according to the monitoring indicators and the exceptional variables to obtain to-be-analyzed data: The step of performing data collection on the target data cluster according to the monitoring indicators and the exceptional variables to obtain to-be-analyzed data specifically includes: processing data information carrying any monitoring indicator or any exceptional variable in the target data cluster (for example, performing information stimulation on the data information carrying any monitoring indicator or any exceptional variable in the target data cluster) to obtain the to-be-analyzed data.
Step 4: grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain exception prediction results: LU501931 The exceptional variables in the to-be-analyzed data are grouped to obtain at least one variable group and exception analysis is performed on the to-be-analyzed data according to the variable group to obtain exception prediction results. By grouping the exceptional variables in the to-be-analyzed data to obtain the variable group, exception analysis can be performed on the to- be-analyzed data by a combination of the exceptional variables to obtain multiple analysis prediction results, thus performing different exception predictions on the target data cluster from multiple dimensions and improving the accuracy of exception prediction for the data clusters.
As a specific embodiment, this step is specifically implemented as follows.
(1) The exceptional variables in the to-be-analyzed data are selected according to a preset selection rule and a variable group is formed of the exceptional variables selected in a same batch. The preset selection rule here is set according to actual needs. For example, a fixed number of variable groups can be obtained by fixed selection of exceptional variables in the to-be-analyzed data in the manner of a preset number; alternatively, the exceptional variables in the to-be-analyzed data are randomly selected in the manner of a preset number and a corresponding number of variable groups can be obtained until all the exceptional variables are selected.
(2) According to the variable group, data corresponding to each exceptional variable in the to-be-analyzed data is obtained respectively to obtain variable data. In this embodiment, according to each exceptional variable in the variable group, data carrying the corresponding exceptional variable in the to-be-analyzed data is obtained respectively and data obtained for the same variable group is integrated to obtain the variable data. In another word, for any variable group, the variable data corresponding to the variable group is obtained by obtaining data carrying exceptional variables of the variable group in the to-be-analyzed data and integrating the data carrying the exceptional variables of the variable group in the to-be-analyzed data.
(3) Exception detection is performed on the variable data respectively to obtain corresponding exception prediction results. The exception detection method for exceptional data is selected from the existing exception detection process according to actual needs. In this embodiment, model training is performed by means of deep learning or a convolutional neural network according to a preset exceptional data sample to obtain an exception detection model. The preset exceptional data sample includes a relationship between each exceptional variable and corresponding data already having an exception. The trained exception detection model cdr/201931 effectively perform exception detection on the input variable data to determine whether the exceptional variable in the variable data is exceptional. When it is determined that the exceptional variable is exceptional, it is determined that an exception may occur in the variable data.
In this embodiment, subsequent to step (3), the data exception analysis method further includes the following steps.
(4) After exception detection is performed on each variable data, if no exceptions occur in all variable data, associated data corresponding to the variable data corresponding to the same variable group is obtained.
(5) Exception analysis is then performed on the obtained associated data to obtain an additional exception prediction result and the additional exception prediction result is added to the foregoing exception prediction results. In the same way as the above-mentioned exception detection for variable data, model training is performed by means of deep learning or a convolutional neural network according to a preset exceptional associated data sample to obtain an association detection model. The preset exceptional associated data sample includes a relationship between data already having an exception corresponding to each exceptional variable and normal data corresponding to any exceptional variable.
Therefore, by obtaining the associated data of the variable data corresponding to the same variable group, exception prediction can be performed on the associated data corresponding to the variable group, thereby improving the reliability of the exception prediction.
In this embodiment, subsequent to step 4, the data exception analysis method further includes a reminder process, and the reminder process is specifically implemented as follows.
(1) Exception notification information corresponding to the target data cluster is obtained and the prediction information in the exception prediction results is matched against notification addresses in the exception notification information.
The exception notification information is obtained first. The exception notification information includes at least one notification address, a corresponding privilege value is set for each notification address, and privilege values corresponding to different notification addresses are different. It should be appreciated that the exception notification information is preset, and each data cluster in the data platform corresponds to the exception notification information.
Exception level evaluation is then performed on the prediction information in the exceptidr501 931 prediction results to obtain corresponding exception values. The exception level evaluation can be performed by using a preset exception level evaluation rule. The preset exception level evaluation rule includes at least two pieces of prediction information and an exception level corresponding to each piece of prediction information, and each exception level corresponds to an exception value. The higher the anomaly level, the higher the corresponding exception value. Correspondingly, the higher the exception value, the higher the degree of exception of the corresponding prediction information.
Finally, the exception value of the prediction information is matched against the privilege values corresponding to the notification addresses to obtain a privilege value corresponding to the exception value and the notification address corresponding to the obtained privilege value is determined.
(ii) The corresponding notification address is reminded of the successfully matched prediction information.
A reminder mode database is preset. The reminder mode database includes at least two privilege values and reminder modes corresponding to the privilege values, and different privilege values correspond to different reminder modes. In this case, a reminder mode corresponding to the obtained privilege value is obtained according to the preset reminder mode database, and a reminder is sent to the corresponding notification address according to the obtained reminder mode.
By obtaining the exception notification information corresponding to the target data cluster and matching the prediction information in the exception prediction results against the notification addresses in the exception notification information, a case where an erroneous reminder is sent to the notification address according to the prediction information can be effectively prevented, thereby improving the accuracy of the prediction information sending.
Step 5: storing the exception prediction results: To store the exception prediction results, in this embodiment, a storage relationship is preset. The preset storage relationship includes the cluster IDs of all the data clusters stored in the data platform and storage areas corresponding to the cluster IDs of all the data clusters. Different cluster identifiers, i.e., different data clusters, correspond to different storage areas. Accordingly, the storage areas need to be configured in advance. It should be appreciated that the cluster ID may 1501931 defined as the storage area name of the corresponding storage area.
According to the cluster ID of the target data cluster and the preset storage relationship, a target storage area corresponding to the target data cluster is determined. Then, the exception prediction result is stored in the target storage area.
By storing the exception prediction results of different data clusters in different storage areas, the reliability of data storage can be improved and subsequent data extraction can be facilitated.
An embodiment of a data exception analysis device: Disclosed is a data exception analysis device, including a memory, a processor, and a computer program stored on the memory and running on the processor, the processor implementing the foregoing data exception analysis method when executing the computer program. Since this method has been described in detail above, it will not be repeated here.

Claims (3)

CLAIMS LU501931
1. À data exception analysis method, comprising: obtaining service components, servers, process information and reference indicators of a target data cluster; obtaining monitoring indicators and exceptional variables corresponding to the service components, servers, process information and reference indicators of the target data cluster according to a preset correspondence; performing data collection on the target data cluster according to the monitoring indicators and the exceptional variables to obtain to-be-analyzed data; grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain exception prediction results; and storing the exception prediction results.
2. The data exception analysis method according to claim 1, wherein prior to the step of obtaining the service components, servers, process information and reference indicators of the target data cluster, the data exception analysis method further comprises: obtaining cluster IDs of all data clusters stored in a data platform; and according to a preset sorting rule and the cluster IDs of the data clusters, sorting the data clusters to obtain a sorting result, so that data exception analysis is performed on the data clusters according to the sorting result.
3. The data exception analysis method according to claim 1, wherein the step of grouping the exceptional variables in the to-be-analyzed data to obtain at least one variable group and performing exception analysis on the to-be-analyzed data according to the variable group to obtain exception prediction results specifically comprises: selecting the exceptional variables in the to-be-analyzed data according to a preset selection rule and forming a variable group of the exceptional variables selected in a same batch; according to the variable group, respectively obtaining data corresponding to each exceptional variable in the to-be-analyzed data to obtain variable data; and performing exception detection on the variable data respectively to obtain corresponding exception prediction results; the step of according to the variable group, respectively obtaining data corresponding to each exceptional variable in the to-be-analyzed data to obtain variable data specifically comprises: according to each exceptional variable in the variable group, obtaining data carrying the corresponding exceptional variable in the to-be-analyzed data respectively and integrating the dad! 501931 obtained for the same variable group to obtain the variable data; subsequent to the step of performing exception detection on the variable data respectively to obtain corresponding exception prediction results, the data exception analysis method further comprises: if no exceptions occur in all the variable data, obtaining associated data corresponding to the variable data corresponding to the same variable group; and performing exception analysis on the associated data to obtain an additional exception prediction result and adding the additional exception prediction result to the exception prediction results.
LU501931A 2022-04-26 2022-04-26 Data exception analysis method and device LU501931B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
LU501931A LU501931B1 (en) 2022-04-26 2022-04-26 Data exception analysis method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
LU501931A LU501931B1 (en) 2022-04-26 2022-04-26 Data exception analysis method and device

Publications (1)

Publication Number Publication Date
LU501931B1 true LU501931B1 (en) 2022-10-26

Family

ID=83852947

Family Applications (1)

Application Number Title Priority Date Filing Date
LU501931A LU501931B1 (en) 2022-04-26 2022-04-26 Data exception analysis method and device

Country Status (1)

Country Link
LU (1) LU501931B1 (en)

Similar Documents

Publication Publication Date Title
CN106992994B (en) Automatic monitoring method and system for cloud service
CN110336838B (en) Account abnormity detection method, device, terminal and storage medium
CN109376873B (en) Operation and maintenance method, operation and maintenance device, electronic equipment and computer readable storage medium
CN112733045B (en) User behavior analysis method and device and electronic equipment
CN113282630B (en) Data query method and device based on interface switching
CN106998336B (en) Method and device for detecting user in channel
CN109558384A (en) Log classification method, device, electronic equipment and storage medium
CN113378899A (en) Abnormal account identification method, device, equipment and storage medium
CN114780606B (en) Big data mining method and system
CN110866249A (en) Method and device for dynamically detecting malicious code and electronic equipment
CN112579603A (en) CDC-based data model dynamic information perception monitoring method and device
CN113254918B (en) Information processing method, electronic device, and computer-readable storage medium
CN114419631A (en) Network management virtual system based on RPA
CN111582722B (en) Risk identification method and device, electronic equipment and readable storage medium
LU501931B1 (en) Data exception analysis method and device
CN115658620B (en) Data authorization sharing method and system based on big data
US20160063394A1 (en) Computing Device Classifier Improvement Through N-Dimensional Stratified Input Sampling
CN111241821B (en) Method and device for determining behavior characteristics of user
CN112052310A (en) Information acquisition method, device, equipment and storage medium based on big data
CN109828902B (en) Interface parameter determining method and device, electronic equipment and storage medium
CN111338958A (en) Parameter generation method and device of test case and terminal equipment
CN111431884A (en) Host computer defect detection method and device based on DNS analysis
CN115037790B (en) Abnormal registration identification method, device, equipment and storage medium
CN110674632A (en) Method and device for determining security level, storage medium and equipment
CN113254919B (en) Abnormal device identification method, electronic device, and computer-readable storage medium

Legal Events

Date Code Title Description
FG Patent granted

Effective date: 20221026