CN112416896A - Data abnormity warning method and device, storage medium and electronic device - Google Patents
Data abnormity warning method and device, storage medium and electronic device Download PDFInfo
- Publication number
- CN112416896A CN112416896A CN202011285361.7A CN202011285361A CN112416896A CN 112416896 A CN112416896 A CN 112416896A CN 202011285361 A CN202011285361 A CN 202011285361A CN 112416896 A CN112416896 A CN 112416896A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- target data
- data
- abnormal
- alarm information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a data abnormity warning method and device, a storage medium and an electronic device. Wherein, the method comprises the following steps: performing exception monitoring on target data in the importing process and the data processing process of the target data; and under the condition that the target data is abnormal, pushing alarm information of the target data. The method and the device solve the technical problem that data abnormity cannot be monitored in the related technology.
Description
Technical Field
The application relates to the field of data monitoring, in particular to a method and a device for alarming data abnormity, a storage medium and an electronic device.
Background
At present, the data quality monitoring schemes in the industry refer to few cases, especially the data monitoring of recommendation systems is almost deficient, a measure capable of performing all-around data monitoring is urgently needed for our system, and the method plays a vital role in solving problems and positioning problems. Based on the situation, the monitoring system is developed.
With the rise of e-commerce, wherein commodity recommendation is a very important link, the system can perform personalized commodity recommendation according to the behaviors of users so as to achieve the effect of recommending thousands of people, but the system follows the monitoring of a more important data link, namely, problems occurring in any link from data generation to data processing to algorithm recommendation and final commodity information display are fatal, so that data quality monitoring of the recommendation system is very important, and a scheme capable of monitoring data abnormity does not exist at present.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the application provides a data abnormity warning method and device, a storage medium and an electronic device, and aims to at least solve the technical problem that data abnormity cannot be monitored in the related art.
According to an aspect of an embodiment of the present application, there is provided a method for alarming data abnormality, including: performing exception monitoring on target data in the importing process and the data processing process of the target data; and under the condition that the target data is abnormal, pushing alarm information of the target data.
Optionally, when the target data is abnormally monitored, monitoring indexes of the target data in multiple dimensions are obtained; and carrying out abnormity monitoring on the target data according to the monitoring indexes on the multiple dimensions.
Optionally, when monitoring indexes of the target data in multiple dimensions are obtained, obtaining a monitoring index in the integrity aspect, where the monitoring index in the integrity aspect is used to monitor whether entity missing, attribute missing, record missing, and field value missing occur; acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard; acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules; acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results; and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
Optionally, when the alarm information that the target data is abnormal is pushed, the corresponding alarm information is pushed according to the abnormal level to which the abnormal type of the target data belongs.
Optionally, when corresponding alarm information is pushed according to the abnormality level to which the abnormality type of the target data belongs, under the condition that the abnormality type of the target data belongs to a first abnormality level, the alarm information is pushed according to a first time interval; and in the case that the abnormality type of the target data belongs to a second abnormality level, pushing alarm information according to a second time interval, wherein the second abnormality level is higher than the first abnormality level, and the second time interval is shorter than the first time interval.
Optionally, after the alarm information that the target data is abnormal is pushed, continuing to monitor whether the target data is abnormal; and under the condition that the target data has the same fault again, reformulating the monitoring index.
Optionally, in the case that the target data is abnormal, the abnormal target data and the fault type of the target data are saved.
According to another aspect of the embodiments of the present application, there is also provided an information abnormality warning device, including: the monitoring unit is used for carrying out exception monitoring on the target data in the importing process and the data processing process of the target data; and the alarm unit is used for pushing alarm information of the abnormity of the target data under the condition that the target data is abnormal.
Optionally, the monitoring unit is further configured to obtain monitoring indexes of the target data in multiple dimensions when performing anomaly monitoring on the target data; and carrying out abnormity monitoring on the target data according to the monitoring indexes on the multiple dimensions.
Optionally, the monitoring unit is further configured to, when monitoring indexes of the target data in multiple dimensions are obtained, obtain a monitoring index in terms of integrity, where the monitoring index in terms of integrity is used to monitor whether entity missing, attribute missing, record missing, and field value missing occur; acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard; acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules; acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results; and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
Optionally, the alarm unit is further configured to, when pushing the alarm information that the target data is abnormal, push corresponding alarm information according to an abnormal level to which an abnormal type of the target data belongs.
Optionally, the alarm unit is further configured to, when pushing corresponding alarm information according to the abnormality level to which the abnormality type of the target data belongs, push alarm information at a first time interval when the abnormality type of the target data belongs to a first abnormality level; and in the case that the abnormality type of the target data belongs to a second abnormality level, pushing alarm information according to a second time interval, wherein the second abnormality level is higher than the first abnormality level, and the second time interval is shorter than the first time interval.
Optionally, the alarm unit is further configured to continue to monitor whether the target data is abnormal after pushing the alarm information that the target data is abnormal; and under the condition that the target data has the same fault again, reformulating the monitoring index.
Optionally, the alarm unit is further configured to, in a case where an abnormality occurs in the target data, save the target data in which the abnormality occurs and a fault type of the target data.
According to another aspect of the embodiments of the present application, there is also provided a storage medium including a stored program which, when executed, performs the above-described method.
According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the above method through the computer program.
In the embodiment of the application, in the importing process and the data processing process of target data, exception monitoring is carried out on the target data; and under the condition that the target data is abnormal, pushing alarm information of the abnormal target data, so that the technical problem that the abnormal data cannot be monitored in the related technology can be solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of an alternative method of alerting of data anomalies according to an embodiment of the present application;
FIG. 2 is a schematic diagram of an alternative data anomaly monitoring scheme according to an embodiment of the present application;
FIG. 3 is a schematic diagram of an alternative data anomaly alerting device according to an embodiment of the present application;
and
fig. 4 is a block diagram of a terminal according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an aspect of the embodiments of the present application, an embodiment of a method for alarming data abnormality is provided. Fig. 1 is a flowchart of an alternative data anomaly alerting method according to an embodiment of the present application, and as shown in fig. 1, the method may include the following steps:
step S1, during the import process and data processing process of the target data, performing anomaly monitoring on the target data.
Optionally, when the target data is abnormally monitored, monitoring indexes of the target data in multiple dimensions are obtained; and carrying out abnormity monitoring on the target data according to the monitoring indexes on the multiple dimensions.
Optionally, when monitoring indexes of the target data in multiple dimensions are obtained, obtaining a monitoring index in the integrity aspect, where the monitoring index in the integrity aspect is used to monitor whether entity missing, attribute missing, record missing, and field value missing occur; acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard; acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules; acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results; and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
The early warning system is a powerful tool for solving the problem that various emergency conditions exist in the processes of data import and ETL (ETL, which is an abbreviation of English Extract-Transform-Load and is used for describing the process of extracting Extract, converting Transform and loading Load from a source end to a destination end of data) to cause final data inconsistency. Therefore, the problem that all links cannot be sensed in the data operation process is solved. The problem of error quick positioning and problem solving are facilitated.
And step S2, when the target data is abnormal, pushing alarm information of the target data.
Optionally, when the alarm information that the target data is abnormal is pushed, the corresponding alarm information is pushed according to the abnormal level to which the abnormal type of the target data belongs.
Optionally, when corresponding alarm information is pushed according to the abnormality level to which the abnormality type of the target data belongs, under the condition that the abnormality type of the target data belongs to a first abnormality level, the alarm information is pushed according to a first time interval; and in the case that the abnormality type of the target data belongs to a second abnormality level, pushing alarm information according to a second time interval, wherein the second abnormality level is higher than the first abnormality level, and the second time interval is shorter than the first time interval.
Optionally, after the alarm information that the target data is abnormal is pushed, continuing to monitor whether the target data is abnormal; and under the condition that the target data has the same fault again, reformulating the monitoring index.
Optionally, in the case that the target data is abnormal, the abnormal target data and the fault type of the target data are saved.
Through the steps, in the importing process and the data processing process of the target data, abnormity monitoring is carried out on the target data; and under the condition that the target data is abnormal, pushing alarm information of the abnormal target data, so that the technical problem that the abnormal data cannot be monitored in the related technology can be solved.
This scheme of adoption can realize that the accuracy detects: verifying whether the result set data is consistent with the source data; data analysis: using the consistency, uniqueness and logicality of the data set to perform statistical analysis and numerical evaluation; and (3) anomaly monitoring: detecting data which are not in accordance with expectation by using a preset algorithm; visual monitoring: displaying the state of data quality by using a control panel; real-time performance: the data quality detection can be carried out in real time, and problems can be found in time; and (3) expandability: may be used for a plurality of data systems. User-friendly page: the system provides a simple and easy-to-use user interface, and can manage data assets and data quality rules; meanwhile, the user can check the data quality result and the user-defined display content through the control panel, so that the final consistency of the data can be ensured, and the recommendation result is real and effective.
As an alternative example, as shown in fig. 2, the following further details the technical solution of the present application with reference to specific embodiments.
S1: the method comprises the steps of analyzing data to be accessed, finding data problems, collecting data quality monitoring requirements, checking the reasonability of the requirements, entering an ETL module if a data access module is not abnormal, mainly cleaning and logically processing the data in the ETL module, and possibly causing the problem that the difference between a data processing result and an expected value is too large in the process, so that specific index monitoring is required according to previous knowledge accumulation and experience.
S2: and formulating the demand into each monitoring index, combing the indexes and determining the indexes.
S3: and applying the indexes to specific monitoring data, correlating the indexes and the data, configuring rules, and formulating a detection range and a detection standard.
The test standard starts from the following points:
integrity: the method mainly comprises four aspects of entity deletion, attribute deletion, record deletion and field value deletion;
simple data statistics: the number of unique or repeated values in a specific column for the statistical table is null, for example, if the number of null records in the statistical field value exceeds a specified threshold, a data loss condition may exist;
the accuracy is as follows: the degree of agreement between a data value and the value set to be accurate, or the difference between the degree of acceptance;
the treatment process is as follows: selecting a source table and a field column; selecting a target table and a target column; the result is calculated by selecting a field comparison rule (greater than, less than, or equal) according to the formula:
rationality: the method mainly comprises the reasonable and effective of format, type, value range and business rule;
the method mainly checks the data, judges whether the data is reasonable under the condition of no storage loss, judges whether the values of partial field data are abnormal, and eliminates or replaces unreasonable data by combining the specific analysis of the service data.
Consistency: data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results; namely, whether the data after the ETL data processing is logically missing or not is ensured, and the consistency before and after the data processing is ensured.
Timeliness: the timeliness and rapidity of the ETL and the application of the data warehouse, and the time consumption, the operation quality and the operation-dependent timeliness of the Jobs. Whether the obtained abnormal data can be timely alarmed or not and whether the normal data can be normally and timely displayed or not are judged, time-consuming response monitoring is achieved, and an early warning prompt can be given out through overtime early warning.
S4: and implementing the configured rules and the formulated check standard into codes, checking the codes, scheduling and executing, configuring and scheduling execution, and periodically executing scheduling to monitor data.
S5: and displaying the problems of the obtained problematic data, classifying the problems, and analyzing the quality of the data to obtain the reasons of the problems.
The problems are divided into the following degrees according to the aspects of operation level, data timeliness, service influence range and the like:
high, low, moderate, very low, and very high severity.
S6: classifying and evaluating the severity of the problem, making a grade, marking an influence range and making a corresponding solution.
For example: the data with higher severity can be monitored in real time to give an early warning for hours, and the data with lower severity can be given an early warning for days or weeks, so that the processing personnel can know which event is processed preferentially, and the events can be ignored.
S7: and (4) implementing the processing scheme, tracking whether the processes processed after the processing are abnormal or not and whether the problems are solved or not, returning to S5 for analysis processing again if the problems cannot be solved, and returning to S1 for analyzing the data and establishing indexes again if the problems are index problems. Thereby forming a data quality closed loop process.
S8: the method has the advantages that the reason and the processing scheme for the problems are summarized, knowledge accumulation and experience accumulation are carried out, and similar problems can be better and faster processed. And for the early warning data uploaded by each module each time, the early warning service can be stored as historical data. Meanwhile, data can be gathered and analyzed in a unified mode, results are returned to the front end and used for displaying reports, developers can conveniently and visually know the data quality, and historical data can be conveniently traced in an abnormal mode.
In the related art, for example, patent No. CN 110400171a proposes a method for analyzing basic data and order data based on an offline store to obtain comparison data, comparing the comparison data with a preset threshold, and triggering an early warning, so as to combine online and offline data to achieve the purpose of early warning a service. But this approach does not address the problem of fine-grained localization.
The invention mainly aims to provide the following advantages: there are many problems in data quality monitoring, such as: the data blood relationship in the data circulation process cannot be monitored. The method has the defects of lack of a monitoring platform with uniform data quality, scattered offline and real-time operation monitoring, insufficient relevance, missing data quality measurement standard, lag data verification and non-uniform data caliber. Data failure handling cannot be closed loop. Data model quality monitoring is lost, and data volume storage resources grow too fast. The system achieves maintenance of data blood relationship based on E-commerce commodity data, unified maintenance of data interface service, monitoring of data volume change, integration of recommendation and monitoring, unified problem handling and standardized service management.
After the project is successfully operated, the quality of the service data is fully ensured. Even if errors occur, the problem can be positioned and solved in the shortest time. Has good monitoring performance and prospective performance.
It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present application.
According to another aspect of the embodiment of the application, a data abnormity warning device for implementing the data abnormity warning method is further provided. Fig. 3 is a schematic diagram of an alternative data anomaly warning device according to an embodiment of the present application, and as shown in fig. 3, the device may include:
the monitoring unit 31 is configured to perform exception monitoring on target data in an import process and a data processing process of the target data; and the alarm unit 33 is configured to, when the target data is abnormal, push alarm information about the abnormality of the target data.
It should be noted that the monitoring unit 31 in this embodiment may be configured to execute step S1 in this embodiment, and the alarm unit 33 in this embodiment may be configured to execute step S2 in this embodiment.
Through the modules, the target data is subjected to exception monitoring in the importing process and the data processing process of the target data; and under the condition that the target data is abnormal, pushing alarm information of the abnormal target data, so that the technical problem that the abnormal data cannot be monitored in the related technology can be solved.
Optionally, the monitoring unit is further configured to obtain monitoring indexes of the target data in multiple dimensions when performing anomaly monitoring on the target data; and carrying out abnormity monitoring on the target data according to the monitoring indexes on the multiple dimensions.
Optionally, the monitoring unit is further configured to, when monitoring indexes of the target data in multiple dimensions are obtained, obtain a monitoring index in terms of integrity, where the monitoring index in terms of integrity is used to monitor whether entity missing, attribute missing, record missing, and field value missing occur; acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard; acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules; acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results; and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
Optionally, the alarm unit is further configured to, when pushing the alarm information that the target data is abnormal, push corresponding alarm information according to an abnormal level to which an abnormal type of the target data belongs.
Optionally, the alarm unit is further configured to, when pushing corresponding alarm information according to the abnormality level to which the abnormality type of the target data belongs, push alarm information at a first time interval when the abnormality type of the target data belongs to a first abnormality level; and in the case that the abnormality type of the target data belongs to a second abnormality level, pushing alarm information according to a second time interval, wherein the second abnormality level is higher than the first abnormality level, and the second time interval is shorter than the first time interval.
Optionally, the alarm unit is further configured to continue to monitor whether the target data is abnormal after pushing the alarm information that the target data is abnormal; and under the condition that the target data has the same fault again, reformulating the monitoring index.
Optionally, the alarm unit is further configured to, in a case where an abnormality occurs in the target data, save the target data in which the abnormality occurs and a fault type of the target data.
It should be noted here that the modules described above are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the disclosure of the above embodiments. It should be noted that the modules as a part of the apparatus may run in a corresponding hardware environment, and may be implemented by software, or may be implemented by hardware, where the hardware environment includes a network environment.
According to another aspect of the embodiment of the application, a server or a terminal for implementing the data abnormality alarming method is further provided.
Fig. 4 is a block diagram of a terminal according to an embodiment of the present application, and as shown in fig. 4, the terminal may include: one or more processors 201 (only one shown), memory 203, and transmission means 205, as shown in fig. 4, the terminal may further comprise an input-output device 207.
The memory 203 may be configured to store software programs and modules, such as program instructions/modules corresponding to the method and apparatus for alarming data abnormality in the embodiment of the present application, and the processor 201 executes various functional applications and data processing by running the software programs and modules stored in the memory 203, that is, implements the method for alarming data abnormality. The memory 203 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 203 may further include memory located remotely from the processor 201, which may be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 205 is used for receiving or sending data via a network, and can also be used for data transmission between a processor and a memory. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 205 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 205 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.
Wherein the memory 203 is specifically used for storing application programs.
The processor 201 may call the application stored in the memory 203 via the transmission means 205 to perform the following steps:
performing exception monitoring on target data in the importing process and the data processing process of the target data;
and under the condition that the target data is abnormal, pushing alarm information of the target data.
The processor 201 is further configured to perform the following steps:
acquiring a monitoring index in the integrity aspect, wherein the monitoring index in the integrity aspect is used for monitoring whether entity missing, attribute missing, record missing and field value missing occur;
acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard;
acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules;
acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results;
and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.
It can be understood by those skilled in the art that the structure shown in fig. 4 is only an illustration, and the terminal may be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), a PAD, etc. Fig. 4 is a diagram illustrating the structure of the electronic device. For example, the terminal may also include more or fewer components (e.g., network interfaces, display devices, etc.) than shown in FIG. 4, or have a different configuration than shown in FIG. 4.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
Embodiments of the present application also provide a storage medium. Alternatively, in this embodiment, the storage medium may be used for a program code of a method for alarming a data abnormality.
Optionally, in this embodiment, the storage medium may be located on at least one of a plurality of network devices in a network shown in the above embodiment.
Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:
performing exception monitoring on target data in the importing process and the data processing process of the target data;
and under the condition that the target data is abnormal, pushing alarm information of the target data.
Optionally, the storage medium is further arranged to store program code for performing the steps of:
acquiring a monitoring index in the integrity aspect, wherein the monitoring index in the integrity aspect is used for monitoring whether entity missing, attribute missing, record missing and field value missing occur;
acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard;
acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules;
acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results;
and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment is not described herein again.
Optionally, in this embodiment, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.
The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, network devices, or the like) to execute all or part of the steps of the method described in the embodiments of the present application.
In the above embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.
Claims (10)
1. An alarm method for data abnormity is characterized by comprising the following steps:
performing exception monitoring on target data in the importing process and the data processing process of the target data;
and under the condition that the target data is abnormal, pushing alarm information of the target data.
2. The method of claim 1, wherein monitoring the target data for anomalies comprises:
acquiring monitoring indexes of the target data on multiple dimensions;
and carrying out abnormity monitoring on the target data according to the monitoring indexes on the multiple dimensions.
3. The method of claim 2, wherein obtaining monitoring metrics of the target data in a plurality of dimensions comprises:
acquiring a monitoring index in the integrity aspect, wherein the monitoring index in the integrity aspect is used for monitoring whether entity missing, attribute missing, record missing and field value missing occur;
acquiring a monitoring index in accuracy, wherein the monitoring index in accuracy is used for monitoring the consistency degree between a data value and a value set as a standard;
acquiring a monitoring index in the aspect of rationality, wherein the monitoring index in the aspect of rationality is used for monitoring the rationality of formats, types, value ranges and business rules;
acquiring monitoring indexes in the aspect of consistency, wherein the monitoring indexes in the aspect of consistency are used for monitoring data difference and consistency of mutual contradiction between systems, uniform definition of service indexes and consistency of data logic processing results;
and acquiring a monitoring index in the timeliness aspect, wherein the monitoring index in the timeliness aspect is used for monitoring the timeliness and rapidity of ETL and application display of the data warehouse, and the running time, the running quality and the running timeliness of the tasks are depended on.
4. The method according to claim 1, wherein pushing alarm information that the target data is abnormal comprises:
and pushing corresponding alarm information according to the abnormal grade of the abnormal type of the target data.
5. The method of claim 4, wherein pushing corresponding alarm information according to the abnormality level to which the abnormality type of the target data belongs comprises:
under the condition that the abnormal type of the target data belongs to a first abnormal grade, alarm information is pushed according to a first time interval;
and in the case that the abnormality type of the target data belongs to a second abnormality level, pushing alarm information according to a second time interval, wherein the second abnormality level is higher than the first abnormality level, and the second time interval is shorter than the first time interval.
6. The method according to claim 1, wherein after the alarm information that the target data is abnormal is pushed, the method further comprises:
continuously monitoring whether the target data is abnormal or not;
and under the condition that the target data has the same fault again, reformulating the monitoring index.
7. The method of claim 1, wherein in case of an anomaly in the target data, the method further comprises:
and saving the target data with the exception and the fault type of the target data.
8. An information abnormality warning device, comprising:
the monitoring unit is used for carrying out exception monitoring on the target data in the importing process and the data processing process of the target data;
and the alarm unit is used for pushing alarm information of the abnormity of the target data under the condition that the target data is abnormal.
9. A storage medium, characterized in that the storage medium comprises a stored program, wherein the program when executed performs the method of any of the preceding claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor executes the method of any of the preceding claims 1 to 7 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011285361.7A CN112416896A (en) | 2020-11-17 | 2020-11-17 | Data abnormity warning method and device, storage medium and electronic device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011285361.7A CN112416896A (en) | 2020-11-17 | 2020-11-17 | Data abnormity warning method and device, storage medium and electronic device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112416896A true CN112416896A (en) | 2021-02-26 |
Family
ID=74831511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011285361.7A Withdrawn CN112416896A (en) | 2020-11-17 | 2020-11-17 | Data abnormity warning method and device, storage medium and electronic device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112416896A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113760882A (en) * | 2021-11-10 | 2021-12-07 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
| CN114064494A (en) * | 2021-11-19 | 2022-02-18 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101105793A (en) * | 2006-07-11 | 2008-01-16 | 阿里巴巴公司 | Data processing method and system of data library |
| CN105871611A (en) * | 2016-03-31 | 2016-08-17 | 浪潮通信信息系统有限公司 | Network management system supervision platform and supervision method |
| CN109947746A (en) * | 2017-10-26 | 2019-06-28 | 亿阳信通股份有限公司 | A kind of quality of data management-control method and system based on ETL process |
| CN111339062A (en) * | 2020-02-24 | 2020-06-26 | 北京达佳互联信息技术有限公司 | Data monitoring method and device, electronic equipment and storage medium |
-
2020
- 2020-11-17 CN CN202011285361.7A patent/CN112416896A/en not_active Withdrawn
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101105793A (en) * | 2006-07-11 | 2008-01-16 | 阿里巴巴公司 | Data processing method and system of data library |
| CN105871611A (en) * | 2016-03-31 | 2016-08-17 | 浪潮通信信息系统有限公司 | Network management system supervision platform and supervision method |
| CN109947746A (en) * | 2017-10-26 | 2019-06-28 | 亿阳信通股份有限公司 | A kind of quality of data management-control method and system based on ETL process |
| CN111339062A (en) * | 2020-02-24 | 2020-06-26 | 北京达佳互联信息技术有限公司 | Data monitoring method and device, electronic equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113760882A (en) * | 2021-11-10 | 2021-12-07 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
| CN113760882B (en) * | 2021-11-10 | 2022-01-28 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
| CN114064494A (en) * | 2021-11-19 | 2022-02-18 | 北京每日菜场科技有限公司 | Data abnormity alarm method and device, electronic equipment and computer readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11657309B2 (en) | Behavior analysis and visualization for a computer infrastructure | |
| CN107025153B (en) | Disk failure prediction method and device | |
| US9817742B2 (en) | Detecting hardware and software problems in remote systems | |
| JP2019502191A (en) | Service call information processing method and device | |
| CN103069749B (en) | The method and system of the isolation of the problem in virtual environment | |
| US20220050765A1 (en) | Method for processing logs in a computer system for events identified as abnormal and revealing solutions, electronic device, and cloud server | |
| CN109960635B (en) | Monitoring and alarming method, system, equipment and storage medium of real-time computing platform | |
| CN111722952A (en) | Fault analysis method, system, equipment and storage medium of business system | |
| CN111445121A (en) | Risk assessment method and apparatus, storage medium, and electronic apparatus | |
| US11115288B2 (en) | Parameter setting method, data analysis device and data analysis system | |
| CN112395156A (en) | Fault warning method and device, storage medium and electronic equipment | |
| CN111984442A (en) | Method and device for detecting abnormality of computer cluster system, and storage medium | |
| AU2020203735A1 (en) | Automated generation and dynamic update of rules | |
| CN112416896A (en) | Data abnormity warning method and device, storage medium and electronic device | |
| CN114844797A (en) | Call chain log and monitoring log association method and micro-service anomaly detection method | |
| CN108665237B (en) | Method for establishing automatic inspection model and positioning abnormity based on business system | |
| US20090113243A1 (en) | Method, Apparatus and Computer Program Product for Rule-Based Directed Problem Resolution for Servers with Scalable Proactive Monitoring | |
| CN114202256A (en) | Architecture upgrading early warning method and device, intelligent terminal and readable storage medium | |
| CN113708986B (en) | Server monitoring apparatus, method and computer-readable storage medium | |
| CN112652351A (en) | Hardware state detection method and device, computer equipment and storage medium | |
| CN117170915A (en) | Data center equipment fault prediction method and device and computer equipment | |
| CN114726649B (en) | Situation awareness evaluation method and device, terminal equipment and storage medium | |
| CN111176950A (en) | Method and equipment for monitoring network card of server cluster | |
| CN115185778A (en) | Database monitoring method and device | |
| CN115543665A (en) | Memory reliability evaluation method and device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210226 |