CN112328619A - Data quality monitoring method, device, system, electronic device and storage medium - Google Patents
Data quality monitoring method, device, system, electronic device and storage medium Download PDFInfo
- Publication number
- CN112328619A CN112328619A CN202011016235.1A CN202011016235A CN112328619A CN 112328619 A CN112328619 A CN 112328619A CN 202011016235 A CN202011016235 A CN 202011016235A CN 112328619 A CN112328619 A CN 112328619A
- Authority
- CN
- China
- Prior art keywords
- configuration
- data
- rule
- quality monitoring
- custom
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06395—Quality analysis or management
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B23/00—Alarms responsive to unspecified undesired or abnormal conditions
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Abstract
The application relates to a data quality monitoring method, a device, a system, an electronic device and a storage medium, wherein the data quality monitoring method comprises the following steps: acquiring configuration information; the configuration information comprises a custom check rule; generating an SQL command and an HTTP interface request according to the configuration information; executing the SQL command and the HTTP interface request to acquire data to be detected from a target database to obtain an execution result; according to the self-defined verification rule, verifying the data to be detected to obtain a verification result; and performing abnormal alarm according to the verification result and the execution result. Through the method and the device, the problem that abnormal data cannot be effectively monitored in the related technology is solved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a system, an electronic apparatus, and a storage medium for monitoring data quality.
Background
Currently, data teams of many companies produce billions of data every day, but do not have a data monitoring and service platform with high data accuracy and a comparison system, which may cause errors in the produced data due to problems of computing logic, systems and the like, and when a downstream enterprise uses the wrong data, decisions of the downstream enterprise are adversely affected, thereby causing huge economic loss of the downstream enterprise.
In the related art, data quality monitoring is performed through a fixedly set data verification rule, however, since types and contents of data produced by different companies are greatly different, the fixedly set data verification rule cannot be applied to all data, which may cause a huge error in a verification result.
At present, an effective solution is not provided for the problem that abnormal data cannot be effectively monitored in the related technology.
Disclosure of Invention
The embodiment of the application provides a data quality monitoring method, a data quality monitoring device, a data quality monitoring system, an electronic device and a storage medium, and at least solves the problem that abnormal data cannot be effectively monitored in the related technology.
In a first aspect, an embodiment of the present application provides a data quality monitoring method, including:
acquiring configuration information; the configuration information comprises a custom check rule;
generating an SQL command and an HTTP interface request according to the configuration information;
executing the SQL command and the HTTP interface request to acquire data to be detected from a target database to obtain an execution result;
according to the self-defined verification rule, verifying the data to be detected to obtain a verification result;
and performing abnormal alarm according to the verification result and the execution result.
In some of these embodiments, the configuration information further includes a custom execution time; the method further comprises the following steps:
acquiring current time, and judging whether the current time is the custom execution time;
and if the current time is the custom execution time, verifying the to-be-detected data according to the custom verification rule to obtain a verification result.
In some embodiments, the threshold parameters in the custom check rule include a regular threshold, a comparand threshold, and a ring rati threshold, wherein:
the regular threshold comprises at least one of a daily regular threshold, a weekly regular threshold, and a monthly regular threshold;
the year of year threshold comprises at least one of a day year threshold, a week year threshold, and a month year threshold;
the ring ratio threshold includes at least one of a daily ring ratio threshold, a weekly ring ratio threshold, and a monthly ring ratio threshold.
In some of these embodiments, the configuration information further includes custom configuration rules, configuration parameters, and fixed parameters; the generating an SQL command and an HTTP interface request according to the configuration information comprises:
generating the SQL command according to the user-defined configuration rule and the configuration parameters;
and generating the HTTP interface request according to the fixed parameters.
In some embodiments, the custom configuration rules include advanced configuration rules and page common configuration rules, where:
if the user-defined configuration rule is the advanced configuration rule, the configuration parameters further comprise an SQL data packet and specified parameters in the SQL data packet;
if the user-defined configuration rule is the common configuration rule of the page, the configuration parameters comprise module configuration, index configuration and rule configuration.
In some embodiments, the configuration mode is the advanced configuration mode; the generating the SQL command according to the custom configuration rule and the configuration parameters comprises:
and replacing the specified parameters in the SQL data packet to generate an executable SQL command.
In some embodiments, the configuration mode is a common configuration mode of the page; the generating the SQL command according to the custom configuration rule and the configuration parameters comprises:
and generating an executable SQL command according to the options in the module configuration, the index configuration and the rule configuration.
In some embodiments, the configuration information further includes custom pre-warning rules; the performing an abnormal alarm according to the verification result and the execution result comprises:
and performing abnormal alarm according to the user-defined early warning rule, the verification result and the execution result.
In some embodiments, the number of the customized early warning rules is multiple; the customized early warning rule comprises a plurality of abnormal warning modes, wherein the abnormal warning modes comprise a daily warning mode and an emergency warning mode.
In a second aspect, an embodiment of the present application provides a data quality monitoring apparatus, including:
the acquisition module is used for acquiring configuration information; the configuration information comprises a custom check rule;
a generating module for generating SQL command and HTTP interface request according to the configuration information
The execution module is used for executing the SQL command and the HTTP interface request so as to acquire the data to be detected from the target database and obtain an execution result;
the verification module is used for verifying the data to be detected according to the self-defined verification rule to obtain a verification result;
and the alarm module is used for carrying out abnormal alarm according to the verification result and the execution result.
In a third aspect, an embodiment of the present application provides a data quality monitoring system, including: the system comprises a user side, a data portal, an LDAP, a monitoring platform, a system database and a target database; wherein:
the LDAP is connected with the user side through the data portal and is used for authenticating login information sent by the user side;
the system database is used for storing configuration information, an execution result and a verification result;
the target database is used for receiving the SQL command and the HTTP interface request sent by the monitoring platform and sending the data to be detected to the monitoring platform;
the monitoring platform is used for executing the data quality monitoring method according to any one of claims 1 to 9.
In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the data quality monitoring method according to the first aspect is implemented.
In a fifth aspect, the present application provides a storage medium, on which a computer program is stored, where the computer program is executed by a processor to implement the data quality monitoring method according to the first aspect.
Compared with the related art, the data quality monitoring method, the data quality monitoring device, the data quality monitoring system, the electronic device and the storage medium provided by the embodiment of the application acquire configuration information; the configuration information comprises a self-defined check rule; generating an SQL command and an HTTP interface request according to the configuration information; executing the SQL command and the HTTP interface request to acquire the data to be detected from the target database to obtain an execution result; according to the self-defined verification rule, verifying the data to be detected to obtain a verification result; and performing abnormal alarm according to the verification result and the execution result, thereby solving the problem that abnormal data cannot be effectively monitored in the related technology.
The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1 is a flow chart of a data quality monitoring method according to an embodiment of the present application;
FIG. 2 is a flowchart illustrating generating an SQL command and an HTTP interface request according to configuration information in an embodiment of the present application;
FIG. 3 is a flow chart of a data quality monitoring method of the preferred embodiment of the present application;
fig. 4 is a block diagram of a hardware structure of a terminal of the data quality monitoring method according to the embodiment of the present application;
fig. 5 is a block diagram of a data quality monitoring system according to an embodiment of the present application;
FIG. 6 is a schematic diagram of anomaly monitoring by a data quality monitoring system in an embodiment of the present application;
fig. 7 is a block diagram of a data quality monitoring apparatus according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.
Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.
Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. Reference herein to "a plurality" means greater than or equal to two. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. Reference throughout this application to the terms "first," "second," "third," and the like, merely distinguish between similar elements and do not denote a particular order of importance.
The various techniques described herein may be applied, but are not limited to, a variety of data quality monitoring devices, platforms, and systems.
Fig. 1 is a flowchart of a data quality monitoring method according to an embodiment of the present application, and as shown in fig. 1, the flowchart includes the following steps:
step S110, obtaining configuration information; the configuration information includes custom verification rules.
The configuration information may be stored in a system database such that the configuration information is read from the system database when the verification task is performed. The configuration information also includes configuration name, belonging service line and database information.
Step S120, generating SQL commands and HTTP interface requests according to the configuration information.
Step S130, SQL commands and HTTP interface requests are executed to acquire the data to be detected from the target database, and execution results are obtained.
Specifically, the SQL command and the HTTP interface request are sent to the target database to acquire the data to be detected from the target database, an execution result is obtained, and if the execution fails, error information is sent to a user corresponding to the target database.
Further, JDBC can be used to execute SQL commands.
JDBC (Java DataBase Connectivity) is a Java API for executing SQL statements, providing uniform access to multiple relational databases, and consists of a set of classes and interfaces written in the Java language. JDBC provides a benchmark by which more advanced tools and interfaces can be built to enable database developers to write database applications.
Further, HTTP interface requests may be performed using the HTTP client toolkit provided by the Apache Software Foundation (ASF).
HTTP live is a child under Apache Jakarta Common that can be used to provide an efficient, up-to-date, feature-rich client programming toolkit that supports the HTTP protocol, and which supports up-to-date versions and recommendations of the HTTP protocol.
And step S140, customizing a verification rule, and verifying the data to be detected to obtain a verification result.
It should be noted that the custom verification rule can be set according to the characteristics of the actual application scenario and the data to be detected, so as to be suitable for different application scenarios and different characteristics of the data to be detected, thereby improving the accuracy of data verification.
According to the practical application scene and the characteristics of the data to be detected, a plurality of or a plurality of user-defined check rules can be adopted to check the data to be detected, and the combination of the plurality of user-defined check rules can also be adopted to check the data to be detected, so that the number and the types of the user-defined check rules are not limited in the embodiment.
And S150, performing abnormal alarm according to the verification result and the execution result.
Acquiring configuration information through the steps S110 to S150; the configuration information comprises a self-defined check rule; generating an SQL command and an HTTP interface request according to the configuration information; executing the SQL command and the HTTP interface request to acquire the data to be detected from the target database to obtain an execution result; defining a calibration rule by user, and calibrating the data to be detected to obtain a calibration result; and performing abnormal alarm according to the checking result and the execution result. By the method and the device, the user-defined check rule can be set according to actual application requirements, so that the data to be detected is checked according to the user-defined check rule, the accuracy of the check result can be improved, and the problem that abnormal data cannot be effectively monitored in the related art is solved. Meanwhile, the data quality monitoring method provided by the application can be suitable for various different application scenes, and is convenient for large-scale application and popularization.
In some of these embodiments, the configuration information further includes a custom execution time; the data quality monitoring method also comprises the following steps:
step (1) obtaining current time, and judging whether the current time is custom execution time;
and (2) if the current time is the custom execution time, verifying the data to be detected according to the custom verification rule to obtain a verification result.
The custom execution time comprises preset execution time values and preset execution time periods of one or more different time dimensions. Wherein the time dimension includes at least one of seconds, minutes, hours, days, weeks, months, and years.
For example, when the time dimension is second, the custom execution time may be set to be executed once per second or once every 30 seconds, or any time period between 1 and 60 may be set to be a preset execution time period, and similarly, the time dimension may be set to be a preset execution time value and a preset execution time period in minutes, hours, days, weeks, months, or years.
It should be noted that, the step of "determining whether the current time is the custom execution time" may be performed before the SQL command and the HTTP interface request are executed, that is, the custom execution time may be determined between the acquisition of the data to be detected. And if the current time is the custom execution time, executing the SQL command and the HTTP interface request to acquire the data to be detected from the target database to obtain an execution result, and further checking the data to be detected according to the custom check rule.
That is to say, the step of "judging whether the current time is the custom execution time" may be executed before the data to be detected is acquired, or the step of "judging whether the current time is the custom execution time" may be executed between the acquisition of the data to be detected and the verification of the data to be detected, which is not limited in this embodiment.
By the embodiment, the current time is obtained, and whether the current time is the custom execution time is judged; if the current time is the self-defined execution time, the data to be detected is verified according to the self-defined verification rule to obtain a verification result, so that the setting of the verification time is more flexible and convenient, and the problem that the verification task cannot be executed at the self-defined time in the prior art is solved.
In some embodiments, the threshold parameters in the custom check rule include a regular threshold, a comparand threshold, and a ring comparand threshold, wherein: the regular threshold comprises at least one of a daily regular threshold, a weekly regular threshold, and a monthly regular threshold; the year-of-year threshold comprises at least one of a day-of-year threshold, a week-of-year threshold, and a month-of-year threshold; the ring ratio threshold includes at least one of a daily ring ratio threshold, a weekly ring ratio threshold, and a monthly ring ratio threshold.
Further, detecting the data to be detected according to a conventional threshold, a same-ratio threshold and a ring-ratio threshold in the custom check rule, and if the current conventional value of the data to be detected is greater than the conventional threshold, determining that the current conventional value is in an abnormal state; if the current same-ratio value of the data to be detected is larger than the same-ratio threshold, determining that the current same-ratio value is in an abnormal state; and if the current ring ratio of the data to be detected is greater than the ring ratio threshold, determining that the current ring ratio is in an abnormal state.
It should be noted that, if any one of the current normal value, the current same ratio value, and the current ring ratio value is in an abnormal state, each abnormal result is recorded and an abnormal alarm is performed according to each abnormal result, or all the abnormal results are summarized and an abnormal alarm is performed according to the summarized abnormal results, which does not limit the specific manner of the abnormal alarm.
Through the embodiment, the threshold parameter in the customized check rule is divided into the conventional threshold, the same-ratio threshold and the ring-ratio threshold, and the conventional threshold, the same-ratio threshold and the ring-ratio threshold are further divided according to the time dimension. According to the embodiment, the threshold parameters in the user-defined check rule are divided more carefully and reasonably, so that the user-defined check rule is more systematic and comprehensive, and the accuracy of the check result can be further improved.
In some embodiments, before obtaining the configuration information, the data quality monitoring method further includes:
(1) configuring second database link information corresponding to the system database; the second database link information comprises a database address, a database name, a database account and a password corresponding to the database account.
(2) The system database is accessed to configure information from the system database based on the second database linking information.
In some embodiments, before the data to be detected is obtained from the target database, the data quality monitoring method further includes:
(1) configuring first database link information corresponding to a target database; the first database link information comprises a database address, a database name, a database account and a password corresponding to the database account.
(2) And accessing the target database according to the first database link information so as to acquire the data to be detected from the target database.
It should be noted that, in the data quality monitoring process, there may be a case of performing verification detection on to-be-detected data in the multiple target databases, and therefore, by configuring the first database link information corresponding to each target database and finding the corresponding target database according to the first database link information, the problem of erroneously reading to-be-detected data when multiple target databases exist can be avoided, and the reliability of the verification result is improved.
In some embodiments, the configuration information further includes custom configuration rules, configuration parameters, and fixed parameters; fig. 2 is a flowchart of generating an SQL command and an HTTP interface request according to configuration information in the embodiment of the present application, and as shown in fig. 2, the flowchart includes the following steps:
step S210, generating an SQL command according to the user-defined configuration rule and the configuration parameters.
Step S220, generating an HTTP interface request according to the fixed parameters.
Generating an SQL command according to the user-defined configuration rule and the configuration parameters through the steps S210 to S220; and generating the HTTP interface request according to the fixed parameters. According to the method and the device, the SQL command and the HTTP interface request are automatically generated according to the user-defined configuration rule, the configuration parameter and the fixed parameter, and technicians do not need to manually configure the SQL command and the HTTP interface request, so that the problem that errors are prone to occurring in manual configuration is solved, the configuration accuracy and the configuration efficiency can be improved, and preparation is made for subsequently accessing the target database according to the SQL command and the HTTP interface request.
In some embodiments, the custom configuration rules include advanced configuration rules and page general configuration rules, wherein:
if the custom configuration rule is a high-level configuration rule, the configuration parameters further include an SQL data packet and specified parameters in the SQL data packet.
If the custom configuration rule is a common configuration rule of the page, the configuration parameters comprise module configuration, index configuration and rule configuration.
It should be noted that the advanced configuration rule is applicable to a case where the configuration staff is a technician, and the technician can perform SQL command assembly and splicing according to the SQL data packet and the specified parameters in the SQL data packet, thereby implementing more complicated SQL command assembly and splicing.
The common configuration rule of the page is suitable for the situation that a configuration person is a common user, the common user can select proper module configuration, index configuration and rule configuration according to the application requirement of the common user, and the processor of the equipment can automatically generate an SQL command according to the selection information of the user, so that the common user is very convenient and easy to operate.
Through the embodiment, two configuration rules, namely the advanced configuration rule and the page common configuration rule, are set in the user-defined configuration rule, so that a configurator (namely a user) can flexibly select a proper configuration rule according to the self condition, the application requirements of two different user groups, namely a common user and a technical staff, can be met simultaneously, the popularization and the application in a large range are facilitated, the application prospect is good, of course, the customized service of the configuration rule can be carried out according to the different user groups, and the satisfaction degree of the user is improved.
In some embodiments, when the configuration mode is the high-level configuration mode, the SQL command can be executed by replacing the specified parameters in the SQL data packet.
The date parameter may be selected as a designated parameter, and other parameters may also be selected as designated parameters, which is not limited in this embodiment.
The configuration format of the SQL command is as follows:
select sum(t.entrance_loss_merchant_num)from dwp_target_funding_eff_m t left join dm_target_org_emp_d o on t.biz_date＝o.biz_date and o.department_id＝1and o.department_type＝'ceo'where t.biz_date＝${bizdate}and t.emp_role＝'cm'。
and the $ bizdate is a custom format of the date parameter, and can be analyzed and replaced according to a custom check rule.
For example, if the threshold parameter in the custom check rule is a daily-to-annular ratio threshold, the date parameter in the SQL data packet is set to be yesterday; and if the threshold parameter in the self-defined check rule is a day comparison threshold, setting the date parameter in the SQL data packet to seven days before.
In some embodiments, when the configuration mode is a common page configuration mode, the executable SQL command is generated according to options in the module configuration, the index configuration, and the rule configuration.
It should be noted that, according to the option of module configuration, index configuration and rule configuration in the selection information submitted by the user, the executable SQL command is automatically generated.
In some embodiments, the configuration information further includes custom pre-warning rules; and performing abnormal alarm according to the self-defined early warning rule, the verification result and the execution result.
Through the embodiment, the user-defined early warning rule is set according to the actual application requirement, and abnormal warning is performed according to the user-defined early warning rule, the verification result and the execution result, so that the method and the device are suitable for various different application scenes and are convenient for large-scale application and popularization.
In some embodiments, the customized early warning rules include a plurality of customized early warning modes; the self-defined early warning mode comprises a nail, a mail box and a short message.
This embodiment has solved the single problem of the mode of reporting an emergency and asking for help or increased vigilance among the prior art through setting up multiple self-defined early warning mode.
In some embodiments, the number of the customized early warning rules is multiple; the customized early warning rule comprises a plurality of abnormal warning modes, wherein the abnormal warning modes comprise a daily warning mode and an emergency warning mode.
It should be noted that, if the abnormal alarm mode is the daily alarm mode, a data quality monitoring report is generated according to the verification result and the execution result within the preset time period. And if the abnormal alarm mode is the emergency alarm mode and the data to be detected is not detected to be in a threshold range corresponding to a certain threshold parameter, performing abnormal alarm aiming at the threshold parameter.
According to the embodiment, various self-defined early warning rules are set, so that a user can conveniently select the early warning rules according to actual application requirements, the early warning rules are suitable for various different application scenes, and large-scale application and popularization are facilitated.
The embodiments of the present application are described and illustrated below by means of preferred embodiments.
Fig. 3 is a flowchart of a data quality monitoring method according to a preferred embodiment of the present application, and as shown in fig. 3, the data quality monitoring method includes the following steps:
step S310, acquiring configuration information; the configuration information comprises a user-defined check rule, a user-defined execution time and a user-defined early warning rule;
step S320, generating an SQL command and an HTTP interface request according to the configuration information;
step S330, SQL commands and HTTP interface requests are executed to acquire data to be detected from a target database to obtain execution results;
step S340, verifying the data to be detected according to the user-defined verification rule and the user-defined execution time to obtain a verification result;
and step S350, performing abnormal alarm according to the self-defined early warning rule, the verification result and the execution result.
It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.
The method provided by the embodiment can be executed in a terminal, a computer or a similar operation device. Taking the operation on the terminal as an example, fig. 4 is a hardware structure block diagram of the terminal of the data quality monitoring method according to the embodiment of the present application. As shown in fig. 4, the terminal 40 may include one or more (only one shown in fig. 4) processors 402 (the processor 402 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 404 for storing data, and optionally may also include a transmission device 406 for communication functions and an input-output device 408. It will be understood by those skilled in the art that the structure shown in fig. 4 is only an illustration and is not intended to limit the structure of the terminal. For example, terminal 40 may also include more or fewer components than shown in FIG. 4, or have a different configuration than shown in FIG. 4.
The memory 404 may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data quality monitoring method in the embodiment of the present application, and the processor 402 executes the computer programs stored in the memory 404 to execute various functional applications and data processing, i.e., implement the method described above. The memory 404 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 404 may further include memory located remotely from the processor 402, which may be connected to the terminal 40 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 406 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the terminal 40. In one example, the transmission device 406 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmitting device 406 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
Fig. 5 is a block diagram of a data quality monitoring system according to an embodiment of the present application, and as shown in fig. 5, the data quality monitoring system 100 includes: user terminal 10, data portal 20, LDAP 30, monitoring platform 40, system database 50 and target database 60, wherein:
the LDAP 30 is connected to the user terminal 10 through the data portal 20, and is used for authenticating login information sent by the user terminal 10;
a system database 50 for storing configuration information, execution results, and verification results;
the target database 60 is used for receiving the SQL command and the HTTP interface request sent by the monitoring platform 40 and sending the data to be detected to the monitoring platform;
the monitoring platform 40 is configured to perform the steps of any of the method embodiments described above.
Further, a system database 50 is connected to the LDAP 30 for storing authentication records generated by the LDAP 30 and interaction information with the user terminal 10.
Further, the system database 50 is connected to the monitoring platform 40, and is further configured to store the execution result, the verification record, the verification result, and the alarm record generated by the monitoring platform 40.
The data quality monitoring system provided herein is further described and illustrated by a specific embodiment.
Fig. 6 is a schematic diagram illustrating anomaly monitoring performed by the data quality monitoring system in the embodiment of the present application, as shown in fig. 6, the data portal 20 is connected to the user side 10, receives login information sent by the user side 10, authenticates the login information, and sends an authentication result to the user side 10. When the login information sent by the user terminal 10 passes the authentication of the data portal 20, the user can access the data in the data portal 20 through the user terminal 10.
The data portal 20 is connected to the LDAP 30, and the user may send a query request for the configuration list to the LDAP 30 through the data portal 20, and the LDAP 30 sends the configuration list to the LDAP 30 after receiving the query request for the configuration list. The user may also render the configuration list on LDAP 30 for easy viewing by the user.
The user sends a configuration request to the LDAP 30 through the user terminal 10 to configure the ID and the parameters on the LDAP 30, and further controls the monitoring platform 40 to acquire the configuration information corresponding to the configuration request from the system database 50 through the LDAP 30 according to the configured ID and the parameters. The monitoring platform 40 generates an SQL command and an HTTP interface request according to the acquired configuration information, and transmits the SQL command and the HTTP interface request to the target database 60 to acquire data to be detected from the target database 60, so as to check the data to be detected according to the customized detection rule, and transmit the check result to the client 10 through the LDAP 30.
The present application further provides a data quality monitoring apparatus, which is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted here. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Fig. 7 is a block diagram of a data quality monitoring apparatus according to an embodiment of the present application, and as shown in fig. 7, the data quality monitoring apparatus 700 includes:
an obtaining module 710, configured to obtain configuration information; the configuration information includes custom verification rules.
And a generating module 720, configured to generate an SQL command and an HTTP interface request according to the configuration information.
The execution module 730 is configured to execute the SQL command and the HTTP interface request to obtain the data to be detected from the target database, and obtain an execution result.
The checking module 740 is configured to check the data to be detected according to the user-defined checking rule to obtain a checking result.
And an alarm module 750, configured to perform an abnormal alarm according to the verification result and the execution result.
In some of these embodiments, the configuration information further includes a custom execution time; the checking module is also used for acquiring the current time and judging whether the current time is the custom execution time; and if the current time is the custom execution time, verifying the data to be detected according to the custom verification rule to obtain a verification result.
In some embodiments, the threshold parameters in the custom check rule include a regular threshold, a comparand threshold, and a ring comparand threshold, wherein: the regular threshold comprises at least one of a daily regular threshold, a weekly regular threshold, and a monthly regular threshold; the year-of-year threshold comprises at least one of a day-of-year threshold, a week-of-year threshold, and a month-of-year threshold; the ring ratio threshold includes at least one of a daily ring ratio threshold, a weekly ring ratio threshold, and a monthly ring ratio threshold.
In some embodiments, the configuration information further includes custom configuration rules, configuration parameters, and fixed parameters; the generating module comprises a first generating unit and a second generating unit, wherein:
and the first generating unit is used for generating the SQL command according to the user-defined configuration rule and the configuration parameters.
And the second generation unit is used for generating the HTTP interface request according to the fixed parameters.
In some embodiments, the custom configuration rules include advanced configuration rules and page general configuration rules, wherein: if the custom configuration rule is a high-level configuration rule, the configuration parameters further comprise an SQL data packet and designated parameters in the SQL data packet; if the custom configuration rule is a common configuration rule of the page, the configuration parameters comprise module configuration, index configuration and rule configuration.
In some embodiments, the first generating unit is further configured to replace the specified parameter in the SQL data packet to generate the executable SQL command if the configuration mode is the high-level configuration mode.
In some embodiments, the first generating unit is further configured to generate an executable SQL command according to options in the module configuration, the index configuration, and the rule configuration if the configuration mode is a common page configuration mode.
In some embodiments, the configuration information further includes a custom early warning rule, and the warning module is further configured to perform an abnormal warning according to the custom early warning rule, the verification result, and the execution result.
In some embodiments, the number of the customized early warning rules is multiple; the customized early warning rule comprises a plurality of abnormal warning modes, wherein the abnormal warning modes comprise a daily warning mode and an emergency warning mode.
The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.
The present embodiment also provides an electronic device comprising a memory having a computer program stored therein and a processor configured to execute the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, acquiring configuration information; the configuration information includes custom verification rules.
And S2, generating an SQL command and an HTTP interface request according to the configuration information.
And S3, executing the SQL command and the HTTP interface request to acquire the data to be detected from the target database and obtain an execution result.
And S4, verifying the data to be detected according to the self-defined verification rule to obtain a verification result.
And S5, performing exception alarm according to the verification result and the execution result.
It should be noted that, for specific examples in this embodiment, reference may be made to examples described in the foregoing embodiments and optional implementations, and details of this embodiment are not described herein again.
In addition, in combination with the data quality monitoring method in the foregoing embodiment, the embodiment of the present application may provide a storage medium to implement. The storage medium having stored thereon a computer program; the computer program, when executed by a processor, implements any of the data quality monitoring methods of the above embodiments.
It should be understood by those skilled in the art that various features of the above-described embodiments can be combined in any combination, and for the sake of brevity, all possible combinations of features in the above-described embodiments are not described in detail, but rather, all combinations of features which are not inconsistent with each other should be construed as being within the scope of the present disclosure.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (13)
1. A method for monitoring data quality, comprising:
acquiring configuration information; the configuration information comprises a custom check rule;
generating an SQL command and an HTTP interface request according to the configuration information;
executing the SQL command and the HTTP interface request to acquire data to be detected from a target database to obtain an execution result;
according to the self-defined verification rule, verifying the data to be detected to obtain a verification result;
and performing abnormal alarm according to the verification result and the execution result.
2. The data quality monitoring method of claim 1, wherein the configuration information further includes a custom execution time; the method further comprises the following steps:
acquiring current time, and judging whether the current time is the custom execution time;
and if the current time is the custom execution time, verifying the to-be-detected data according to the custom verification rule to obtain a verification result.
3. The data quality monitoring method of claim 1, wherein the threshold parameters in the custom check rule include a regular threshold, a same-ratio threshold, and a ring-ratio threshold, wherein:
the regular threshold comprises at least one of a daily regular threshold, a weekly regular threshold, and a monthly regular threshold;
the year of year threshold comprises at least one of a day year threshold, a week year threshold, and a month year threshold;
the ring ratio threshold includes at least one of a daily ring ratio threshold, a weekly ring ratio threshold, and a monthly ring ratio threshold.
4. The data quality monitoring method of claim 1, wherein the configuration information further comprises custom configuration rules, configuration parameters, and fixed parameters; the generating an SQL command and an HTTP interface request according to the configuration information comprises:
generating the SQL command according to the user-defined configuration rule and the configuration parameters;
and generating the HTTP interface request according to the fixed parameters.
5. The data quality monitoring method of claim 4, wherein the custom configuration rules include advanced configuration rules and page common configuration rules, wherein:
if the user-defined configuration rule is the advanced configuration rule, the configuration parameters further comprise an SQL data packet and specified parameters in the SQL data packet;
if the user-defined configuration rule is the common configuration rule of the page, the configuration parameters comprise module configuration, index configuration and rule configuration.
6. The data quality monitoring method according to claim 5, wherein the configuration mode is the advanced configuration mode; the generating the SQL command according to the custom configuration rule and the configuration parameters comprises:
and replacing the specified parameters in the SQL data packet to generate an executable SQL command.
7. The data quality monitoring method according to claim 5, wherein the configuration mode is a common configuration mode of the page; the generating the SQL command according to the custom configuration rule and the configuration parameters comprises:
and generating an executable SQL command according to the options in the module configuration, the index configuration and the rule configuration.
8. The data quality monitoring method of claim 1, wherein the configuration information further comprises custom pre-warning rules; the performing an abnormal alarm according to the verification result and the execution result comprises:
and performing abnormal alarm according to the user-defined early warning rule, the verification result and the execution result.
9. The data quality monitoring method according to claim 8, wherein the number of the customized early warning rules is multiple; the customized early warning rule comprises a plurality of abnormal warning modes, wherein the abnormal warning modes comprise a daily warning mode and an emergency warning mode.
10. A data quality monitoring apparatus, comprising:
the acquisition module is used for acquiring configuration information; the configuration information comprises a custom check rule;
a generating module for generating SQL command and HTTP interface request according to the configuration information
The execution module is used for executing the SQL command and the HTTP interface request so as to acquire the data to be detected from the target database and obtain an execution result;
the verification module is used for verifying the data to be detected according to the self-defined verification rule to obtain a verification result;
and the alarm module is used for carrying out abnormal alarm according to the verification result and the execution result.
11. A data quality monitoring system, comprising: the system comprises a user side, a data portal, an LDAP, a monitoring platform, a system database and a target database; wherein:
the LDAP is connected with the user side through the data portal and is used for authenticating login information sent by the user side;
the system database is used for storing configuration information, an execution result and a verification result;
the target database is used for receiving the SQL command and the HTTP interface request sent by the monitoring platform and sending the data to be detected to the monitoring platform;
the monitoring platform is used for executing the data quality monitoring method according to any one of claims 1 to 9.
12. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is arranged to run the computer program to perform the data quality monitoring method of any one of claims 1 to 9.
13. A storage medium, in which a computer program is stored, wherein the computer program is arranged to perform the data quality monitoring method of any one of claims 1 to 9 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011016235.1A CN112328619A (en) | 2020-09-24 | 2020-09-24 | Data quality monitoring method, device, system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011016235.1A CN112328619A (en) | 2020-09-24 | 2020-09-24 | Data quality monitoring method, device, system, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112328619A true CN112328619A (en) | 2021-02-05 |
Family
ID=74304025
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011016235.1A Pending CN112328619A (en) | 2020-09-24 | 2020-09-24 | Data quality monitoring method, device, system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112328619A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113032250A (en) * | 2021-03-11 | 2021-06-25 | 中国工商银行股份有限公司 | Automatic calibration system and method for environmental parameters |
| CN113128848A (en) * | 2021-04-01 | 2021-07-16 | 广州虎牙科技有限公司 | Data quality monitoring method for full-service index, electronic equipment and storage medium |
| CN113127471A (en) * | 2021-05-11 | 2021-07-16 | 数库（上海）科技有限公司 | Method, device, equipment and storage medium for automatic data quality inspection |
| CN113377751A (en) * | 2021-06-02 | 2021-09-10 | 浪潮软件股份有限公司 | Integrated data comparison method |
| CN113641628A (en) * | 2021-08-13 | 2021-11-12 | 中国联合网络通信集团有限公司 | Data quality detection method, device, equipment and storage medium |
| CN115496648A (en) * | 2022-11-16 | 2022-12-20 | 摩尔线程智能科技(北京)有限责任公司 | Management method, management device and management system of graphics processor |
| CN116055350A (en) * | 2023-01-03 | 2023-05-02 | 重庆长安汽车股份有限公司 | Json-based data communication quality detection method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010138412A1 (en) * | 2009-05-29 | 2010-12-02 | Invensys Systems, Inc. | Control configuration with control objects that are fieldbus protocol-aware and that self-define tracked parameters |
| WO2017101606A1 (en) * | 2015-12-15 | 2017-06-22 | 北京奇虎科技有限公司 | System and method for collecting and analyzing data |
| CN109032824A (en) * | 2018-05-31 | 2018-12-18 | 康键信息技术（深圳）有限公司 | Database method of calibration, device, computer equipment and storage medium |
| CN110245049A (en) * | 2019-06-06 | 2019-09-17 | 深圳前海微众银行股份有限公司 | Monitoring method, device, equipment and the storage medium of product configuration data |
| CN110457371A (en) * | 2019-08-13 | 2019-11-15 | 杭州有赞科技有限公司 | Data managing method, device, storage medium and system |
-
2020
- 2020-09-24 CN CN202011016235.1A patent/CN112328619A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010138412A1 (en) * | 2009-05-29 | 2010-12-02 | Invensys Systems, Inc. | Control configuration with control objects that are fieldbus protocol-aware and that self-define tracked parameters |
| WO2017101606A1 (en) * | 2015-12-15 | 2017-06-22 | 北京奇虎科技有限公司 | System and method for collecting and analyzing data |
| CN109032824A (en) * | 2018-05-31 | 2018-12-18 | 康键信息技术（深圳）有限公司 | Database method of calibration, device, computer equipment and storage medium |
| CN110245049A (en) * | 2019-06-06 | 2019-09-17 | 深圳前海微众银行股份有限公司 | Monitoring method, device, equipment and the storage medium of product configuration data |
| CN110457371A (en) * | 2019-08-13 | 2019-11-15 | 杭州有赞科技有限公司 | Data managing method, device, storage medium and system |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113032250A (en) * | 2021-03-11 | 2021-06-25 | 中国工商银行股份有限公司 | Automatic calibration system and method for environmental parameters |
| CN113128848A (en) * | 2021-04-01 | 2021-07-16 | 广州虎牙科技有限公司 | Data quality monitoring method for full-service index, electronic equipment and storage medium |
| CN113127471A (en) * | 2021-05-11 | 2021-07-16 | 数库（上海）科技有限公司 | Method, device, equipment and storage medium for automatic data quality inspection |
| CN113377751A (en) * | 2021-06-02 | 2021-09-10 | 浪潮软件股份有限公司 | Integrated data comparison method |
| CN113641628A (en) * | 2021-08-13 | 2021-11-12 | 中国联合网络通信集团有限公司 | Data quality detection method, device, equipment and storage medium |
| CN113641628B (en) * | 2021-08-13 | 2023-06-16 | 中国联合网络通信集团有限公司 | Data quality detection method, device, equipment and storage medium |
| CN115496648A (en) * | 2022-11-16 | 2022-12-20 | 摩尔线程智能科技(北京)有限责任公司 | Management method, management device and management system of graphics processor |
| CN116055350A (en) * | 2023-01-03 | 2023-05-02 | 重庆长安汽车股份有限公司 | Json-based data communication quality detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112328619A (en) | Data quality monitoring method, device, system, electronic device and storage medium | |
| US11102081B1 (en) | Quantitative network testing framework for 5G and subsequent generation networks | |
| CN111143163B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN110908909B (en) | Automatic test method, device, storage medium and equipment | |
| CN109165170B (en) | Method and system for automatic request test | |
| CN112039900B (en) | Network security risk detection method, system, computer device and storage medium | |
| CN111611021B (en) | Log data transmission method, device, computer equipment and storage medium | |
| CN111258900A (en) | Interface test script generation method, system, equipment and storage medium | |
| CN109218407B (en) | Code management and control method based on log monitoring technology and terminal equipment | |
| CN111106899B (en) | Data verification method and device in Internet of things, computer equipment and storage medium | |
| CN109639724B (en) | Password retrieving method, password retrieving device, computer device and storage medium | |
| CN104869019A (en) | Feedback method and apparatus for positioning faults, and server | |
| CN111970236B (en) | Cross-network data transmission method and device | |
| CN106339273A (en) | Application program restoration method, terminal and server | |
| CN111767173A (en) | Network equipment data processing method and device, computer equipment and storage medium | |
| CN111723019A (en) | Interface debugging method and system | |
| CN107483283A (en) | Communicate method for testing reliability and device | |
| WO2022033079A1 (en) | Pico base station configuration method and apparatus, and storage medium and electronic apparatus | |
| CN115730935A (en) | Data processing method, device and equipment based on block chain and readable storage medium | |
| CN112184166A (en) | Data reporting method, device, equipment and readable storage medium of business system | |
| CN113434405B (en) | Method and device for determining test file, storage medium and electronic device | |
| CN113467782B (en) | Method, device and equipment for determining business process | |
| CN115622890A (en) | Data configuration method and device, computer equipment and storage medium | |
| CN114327981A (en) | Safety verification system, method and device of function safety mechanism | |
| CN111694752B (en) | Application testing method, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210205 |
|
| RJ01 | Rejection of invention patent application after publication |