CN114357032A - Data quality monitoring method and device, electronic equipment and storage medium - Google Patents
Data quality monitoring method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114357032A CN114357032A CN202210013830.2A CN202210013830A CN114357032A CN 114357032 A CN114357032 A CN 114357032A CN 202210013830 A CN202210013830 A CN 202210013830A CN 114357032 A CN114357032 A CN 114357032A
- Authority
- CN
- China
- Prior art keywords
- target
- verification
- rule
- target data
- engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The application discloses a data quality monitoring method and device, electronic equipment and a storage medium, belongs to the technical field of data processing, and is used for solving the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring and realizing that a check task simultaneously supports a plurality of data source engines. The method comprises the following steps: determining a target check rule; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a data quality monitoring method and device, electronic equipment and a storage medium.
Background
With the advent of the big data age, the application of data is ubiquitous. The data quality is used as a basis for the effectiveness and accuracy of data analysis and data mining conclusions, and the importance of the data quality is self-evident. The data quality monitoring is to periodically check and verify the original data, the intermediate data and the result data of the data task so as to ensure the correctness or rationality of the output result of the data task.
In reality, data to be monitored may be stored in different data engines, that is, data of multiple data sources needs to be monitored. The method comprises the steps of designing a set of data checking tasks for each data source, but the method only supports a single calculation engine and cannot realize migration and multiplexing on other engines, and writing data results in multiple data sources into one data source through tasks through additional tasks and checking the data source under the same engine, so that redundancy exists in the data engines, and storage space and network transmission are wasted.
Disclosure of Invention
The embodiment of the application provides a data quality monitoring method and device, electronic equipment and a storage medium, which can solve the problems of storage space waste and network transmission waste in data quality monitoring of multiple data sources, and realize that one-time verification task supports multiple data source engines at the same time.
In a first aspect, an embodiment of the present application provides a data quality monitoring method, where the method includes: determining a target check rule; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
In a second aspect, an embodiment of the present application provides a data quality monitoring apparatus, where the apparatus includes: the first determining module is used for determining a target verification rule; a second determination module for determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources; and the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or an instruction stored on the memory and executable on the processor, where the program or the instruction, when executed by the processor, implements the steps of the data quality monitoring method according to the first aspect.
In a fourth aspect, the present application provides a readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the data quality monitoring method according to the first aspect.
In a fifth aspect, an embodiment of the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the steps of the data quality monitoring method according to the first aspect.
In the embodiment of the application, a target verification rule is determined; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
Drawings
Fig. 1 is a schematic flowchart of a data quality monitoring method according to an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram of another data quality monitoring method provided in the embodiments of the present application;
FIG. 3 is a schematic flow chart diagram of another data quality monitoring method provided by the embodiments of the present application;
fig. 4 is a schematic structural diagram of a data quality monitoring apparatus according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of another data quality monitoring apparatus provided in an embodiment of the present application;
FIG. 6 is a schematic structural diagram of another data quality monitoring apparatus provided in an embodiment of the present application;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present application;
fig. 8 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
A data quality monitoring method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios thereof.
Fig. 1 illustrates a data quality monitoring method provided by an embodiment of the present invention, which may be performed by an electronic device that may include at least one of the following: servers, server clusters, Network Attached Storage (NAS), Personal Computers (PCs). In other words, the method may be performed by software or hardware installed in the electronic device, the method comprising the steps of:
s102: and determining a target verification rule.
The step is used for determining a target verification rule, the target verification rule is used for verifying the integrity, consistency, accuracy and the like of data, and the target verification rule is a basic step in the data quality monitoring process.
S104: determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources.
The method comprises the step of determining a target data source engine corresponding to a target data source, wherein the target data source engine is used for linking one or more target data sources. And the data to be monitored is stored in the one or more target data sources, and the target data source engine comprises information such as a data source engine type, a data source connection mode, a user name and a password. In this step, Java Database connection (JDBC) may be used to Support multiple data sources, such as Amazon cloud Storage (Amazon Simple Storage Service, Amazon S3), Distributed File System (HDFS), relational Database System (PostgreSQL, MySQL, and columnar Storage Database System (Clickhouse), or Operation Support System (Operation Support Systems, OSS) that may Support telecommunication Service development and Operation, and easily extensible fully hosted PB grade data warehouse Service System (Amazon Redshift), depending on whether Storage corresponding to the engine supports JDBC connection.
S106: and according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources.
According to the target verification rule, the one or more target data sources contained in the target data source engine are linked in the data source link mode, the target data are verified, and a target verification result is determined. In the step of consistency check, mutual check between multiple data sources is supported, for example, md5 encoding data in a target data table (source table) and a source data table (target table) in a row latitude may be performed and then one-to-one comparison may be performed to obtain a consistency check result.
S108: and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
Corresponding to the target verification rule, judging whether the target verification result meets the preset verification value according to the preset verification rule, and obtaining a passed or failed verification result. For example, this step may be used to determine whether the check, such as the maximum, minimum, and number, meets the legal range, belongs to a single rule check, where the length of the returned result is 1 and only one value matches, or is used to verify the number of illegal records, belongs to a matching check of the number in the single rule check, or is used to compare the consistency of the returned result, belongs to a multi-rule check, where the sequence of the fields that need to be returned is consistent, and if no sequence field is configured, the returned result is fully sorted, or is used to verify the number of illegal results returned after the two comparison rules are associated, and belongs to a multi-rule check, and if the number is greater than 0, the returned result does not pass.
In the data quality monitoring method provided by the embodiment of the invention, a target verification rule is determined; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
Fig. 2 illustrates another data quality monitoring method provided by an embodiment of the invention, which may be performed by an electronic device that may include at least one of: server, server cluster, NAS, PC. In other words, the method may be performed by software or hardware installed in the electronic device, the method comprising the steps of:
s201: and establishing a check rule database, wherein the check rule database is used for storing a plurality of rule configuration information and a plurality of engine configuration information.
For example, in this step, a relational database may be used to store rules for verification, such as MySQL, to design a system database model, and the generated data table structure includes information such as verification type, data association manner, engine configuration, rule type, rule configuration, execution record, and the like. The check rule type may be a single rule or a multi-rule check, where a single rule may include: null detection, primary key detection, table row number detection, average detection, sum detection, maximum detection, minimum detection, regular expression detection, numerical format detection, enumerated value detection, numerical range detection, identity card verification, custom rules, null string detection, and the like. The single rule checking task configuration table comprises information such as a rule identity identification number, a checking task, a target data table, a checking field, a filter, error result matching, a result matching identification number, an engine identification number, a table owner identification number and remarks. The multi-rule checking task configuration table comprises information such as rule 1 detailed information identification numbers, rule 2 detailed information identification numbers, connection identification numbers, result matching identification numbers, specific connection information, filters, owner identification numbers and remarks. The data association mode may include full connection, external connection, left connection, right connection, and the like. The data source engine configuration table includes an engine type, a Uniform Resource Locator (URL) connection mode, a user name, a password, and the like.
S202: and reading a data table in the check rule database to obtain target rule configuration information and target engine configuration information.
This step is used to read target rule configuration information and target engine configuration information from the check rule database, for example, target rule configuration information such as rule identification number, check task, target data table, check field, filter, error result matching, result matching identification number, engine identification number, table owner identification number, remark, etc. in the single-rule check task configuration table, and/or engine configuration information such as engine type, URL connection mode, user name, password, etc. in the data source engine configuration table.
S203: and determining a target verification rule according to the target rule configuration information.
In this step, a target verification rule is determined by using a predetermined computer programming language according to the target rule configuration information. For example, in this step, a computer programming Language python may be used to read the rule types and parameters in the database, and the rule types and parameters are spliced into an executable Structured Query Language (SQL) corresponding to the target check rule, which is referred to as rule SQL for short.
S204: and determining a target data source engine according to the target engine configuration information.
And determining a target data source engine according to the target engine configuration information. Wherein the target data source engine is configured to link one or more target data sources. And the data to be monitored is stored in the one or more target data sources, and the target data source engine comprises data source engine configuration information, a data source link mode and the like. For example, the step may use an engine configured by JDBC links, and may also implement an engine supporting multiple data engines, such as Facebook open-source distributed SQL query engine Presto, a relational database system engine, such as PostgreSQL, MySQL, Hadoop data warehouse tool Hive, a fast general computing engine Apache Spark specially designed for large-scale data processing, and the like.
S206: and according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources.
The step may be described in step S106 in the embodiment of fig. 1, and repeated descriptions of the repeated parts are omitted here.
S208: and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
This step can adopt the description of step S108 in the embodiment of fig. 1, and the repeated parts are not described herein again.
In the data quality monitoring method provided by the embodiment of the invention, a target verification rule is determined; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
According to the data quality monitoring method provided by the embodiment of the invention, a check rule database is established, and the check rule database is used for storing a plurality of rule configuration information and a plurality of engine configuration information; reading a data table in the check rule database to obtain target rule configuration information and target engine configuration information; determining a target verification rule according to the target rule configuration information; determining a target data source engine according to the target engine configuration information, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
Fig. 3 illustrates another data quality monitoring method provided by an embodiment of the invention, which may be performed by an electronic device that may include at least one of: server, server cluster, NAS, PC. In other words, the method may be performed by software or hardware installed in the electronic device, the method comprising the steps of:
s302: and determining a target verification rule.
The step may be described in step S102 in the embodiment of fig. 1, and repeated descriptions of the repeated parts are omitted here.
S304: determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources.
The step may be described in step S104 in the embodiment of fig. 1, and repeated descriptions of the repeated parts are omitted here.
S305: and establishing corresponding engine links according to the target data source engine, and reading target data from the one or more target data sources.
The step is used for establishing a data source engine link according to the target data source engine and determining target data needing to be monitored, wherein the target data come from one or more target data sources. For example, in this step, an engine connection instance may be created for the engine configured by JDBC link, and the target data may be read from different storage systems corresponding to the one or more data sources according to the engine reading and writing method.
S306: and according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources.
The step may be described in step S106 in the embodiment of fig. 1, and repeated descriptions of the repeated parts are omitted here.
S308: and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
This step can adopt the description of step S108 in the embodiment of fig. 1, and the repeated parts are not described herein again.
In an implementation manner, the present embodiment may further include step S301, and step S301 may adopt the description of step S201 in the embodiment of fig. 2, and repeated descriptions of repeated parts are omitted here.
In one implementation manner, the embodiment may further include a step of performing batch verification tasks at a predetermined time by using a scheduling engine before step S301.
In one implementation manner, the present embodiment may further include step S310: and sending alarm information under the condition that the verification result is failed.
In the data quality monitoring method provided by the embodiment of the invention, a target verification rule is determined; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
In the data quality monitoring method provided by the embodiment of the invention, a target verification rule is determined; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; establishing corresponding engine links according to the target data source engine, and reading target data from the one or more target data sources; according to the target verification rule and the target data source engine, verifying the target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
It should be noted that, in the data quality monitoring method provided in the embodiment of the present application, the execution subject may be a data quality monitoring apparatus, or a control module in the data quality monitoring apparatus for executing the data quality monitoring method. In the embodiment of the present application, a data quality monitoring apparatus is taken as an example to execute the data quality monitoring method, and a data quality monitoring apparatus provided in the embodiment of the present application is described.
Fig. 4 is a schematic structural diagram of a data quality monitoring apparatus according to an embodiment of the present invention. As shown in fig. 4, a data quality monitoring apparatus 400 includes: a first determination module 402, a second determination module 404, a verification module 406, and a verification module 408.
A first determining module 402, configured to determine a target verification rule.
A second determining module 404, configured to determine a target data source engine, where the target data source engine is configured to link one or more target data sources.
A checking module 406, configured to check target data according to the target checking rule and the target data source engine, and determine a target checking result, where the target data is from the one or more target data sources.
The verifying module 408 is configured to verify the target verification result according to a predetermined verification rule to obtain a verification result, where the predetermined verification rule includes a predetermined verification value and a predetermined verification type.
The data quality monitoring device provided by the embodiment of the invention is used for determining a target verification rule through a first determining module; a second determination module for determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources; and the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, so that the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
Fig. 5 is a schematic structural diagram of another data quality monitoring apparatus according to an embodiment of the present invention. As shown in fig. 5, a data quality monitoring apparatus 500 includes: a first establishing module 501, a first determining module 502, a second determining module 504, a verifying module 506 and a verifying module 508.
A first establishing module 501, configured to establish a check rule database, where the check rule database is configured to store a plurality of rule configuration information and a plurality of engine configuration information.
A first determining module 502, configured to read a data table in the check rule database to obtain target rule configuration information; and determining a target verification rule according to the target rule configuration information.
A second determining module 504, configured to read a data table in the check rule database to obtain target engine configuration information; and determining a target data source engine according to the target engine configuration information.
A checking module 506, configured to check target data according to the target checking rule and the target data source engine, and determine a target checking result, where the target data is from the one or more target data sources.
The verifying module 508 is configured to verify the target verification result according to a predetermined verification rule to obtain a verification result, where the predetermined verification rule includes a predetermined verification value and a predetermined verification type.
The data quality monitoring device provided by the embodiment of the invention is used for determining a target verification rule through a first determining module; a second determination module for determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources; and the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, so that the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
The data quality monitoring device provided by the embodiment of the invention is used for establishing a verification rule database through a first establishing module, wherein the verification rule database is used for storing a plurality of rule configuration information and a plurality of engine configuration information; the first determining module is configured to: reading a data table in the check rule database to obtain target rule configuration information; determining a target verification rule according to the target rule configuration information; the second determining module is configured to: reading a data table in the check rule database to obtain target engine configuration information; determining a target data source engine according to the target engine configuration information; the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources; and the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, so that the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
Fig. 6 is a schematic structural diagram of a data quality monitoring apparatus according to an embodiment of the present invention. As shown in fig. 6, a data quality monitoring apparatus 600 includes: a first establishing module 601, a first determining module 602, a second determining module 604, a second establishing module 605, a verifying module 606 and a verifying module 608.
The first establishing module 601 is configured to establish a check rule database, where the check rule database is configured to store a plurality of rule configuration information and a plurality of engine configuration information.
A first determining module 602, configured to read a data table in the check rule database to obtain target rule configuration information; and determining a target verification rule according to the target rule configuration information.
A second determining module 604, configured to read a data table in the check rule database to obtain target engine configuration information; and determining a target data source engine according to the target engine configuration information.
A second establishing module 605, configured to establish a corresponding engine link according to the target data source engine, and read the target data from the one or more target data sources.
A checking module 606, configured to check target data according to the target checking rule and the target data source engine, and determine a target checking result, where the target data is from the one or more target data sources.
The verifying module 608 is configured to verify the target verification result according to a predetermined verification rule to obtain a verification result, where the predetermined verification rule includes a predetermined verification value and a predetermined verification type.
And the alarm module 610 is configured to send alarm information if the verification result is that the verification result is not passed.
In one implementation, the apparatus may further include: and the scheduling module is used for executing the verification tasks in batches at preset time by using the scheduling engine.
The data quality monitoring device provided by the embodiment of the invention is used for determining a target verification rule through a first determining module; a second determination module for determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources; and the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type, so that the problems of storage space waste and network transmission waste in multi-data-source data quality monitoring can be solved, and a single verification task can simultaneously support a plurality of data source engines.
The data quality monitoring device provided by the embodiment of the invention is used for establishing a verification rule database through a first establishing module, wherein the verification rule database is used for storing a plurality of rule configuration information and a plurality of engine configuration information; the first determining module is configured to: reading a data table in the check rule database to obtain target rule configuration information; determining a target verification rule according to the target rule configuration information; the second determining module is configured to: reading a data table in the check rule database to obtain target engine configuration information; determining a target data source engine according to the target engine configuration information; the second establishing module is used for establishing corresponding engine links according to the target data source engine and reading the target data from the one or more target data sources; the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources; the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type; and the alarm module is used for sending alarm information under the condition that the verification result is not passed, solving the problems of storage space waste and network transmission waste existing in the data quality monitoring of multiple data sources, and realizing that one-time verification task supports multiple data source engines simultaneously.
It should be noted that the embodiment of the data quality monitoring apparatus in this specification and the embodiment of the data quality monitoring method in this specification are based on the same inventive concept, and therefore, for specific implementation of the embodiment of the data quality monitoring apparatus, reference may be made to the implementation of the corresponding embodiment of the data quality monitoring method, and repeated details are not repeated.
The data quality monitoring device in the embodiment of the present application may be a device, or may be a component, an integrated circuit, or a chip in a terminal. The device can be mobile electronic equipment or non-mobile electronic equipment. By way of example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine or a self-service machine, and the like, and the embodiments of the present application are not particularly limited.
The data quality monitoring device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, and embodiments of the present application are not limited specifically.
The data quality monitoring device provided in the embodiment of the present application can implement each process implemented in the method embodiments of fig. 1 to fig. 3, and is not described here again to avoid repetition.
Optionally, as shown in fig. 7, an electronic device 700 is further provided in this embodiment of the present application, and includes a processor 701, a memory 702, and a program or an instruction stored in the memory 702 and executable on the processor 701, where the program or the instruction is executed by the processor 701 to implement each process of the data quality monitoring method embodiment, and can achieve the same technical effect, and no further description is provided here to avoid repetition.
It should be noted that the electronic device in the embodiment of the present application includes the mobile electronic device and the non-mobile electronic device described above.
Drawing (A)8The hardware structure diagram of the electronic device is used for realizing the embodiment of the application.
The electronic device 800 includes, but is not limited to: a radio frequency unit 801, a network module 802, an audio output unit 803, an input unit 804, a sensor 805, a display unit 806, a user input unit 807, an interface unit 808, a memory 809, and a processor 810.
Those skilled in the art will appreciate that the electronic device 800 may further comprise a power source (e.g., a battery) for supplying power to the various components, and the power source may be logically connected to the processor 810 via a power management system, so as to manage charging, discharging, and power consumption management functions via the power management system. The electronic device structures shown in the figures do not constitute limitations of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is not repeated here.
The processor 810 is configured to determine a target verification rule; determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources; according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources; and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
In one implementation, the processor 810 is configured to establish a check rule database, where the check rule database is configured to store a plurality of rule configuration information and a plurality of engine configuration information; the determining of the target verification rule comprises: reading a data table in the check rule database to obtain target rule configuration information; determining a target verification rule according to the target rule configuration information; the target data source determining engine comprises: reading a data table in the check rule database to obtain target engine configuration information; and determining a target data source engine according to the target engine configuration information.
In one implementation, the processor 810 is configured to establish a corresponding engine link according to the target data source engine, and read the target data from the one or more target data sources; and the alarm information is sent out under the condition that the verification result is failed.
The embodiment can realize each process of the data quality monitoring method embodiment, and can achieve the same technical effect, and for avoiding repetition, the details are not repeated here. It should be understood that in the embodiment of the present application, the input Unit 804 may include a Graphics Processing Unit (GPU) 8041 and a microphone 8042, and the Graphics Processing Unit 8041 processes image data of a still picture or a video obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The display unit 806 may include a display panel 8061, and the display panel 8061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 807 includes a touch panel 8071 and other input devices 8072. A touch panel 8071, also referred to as a touch screen. The touch panel 8071 may include two portions of a touch detection device and a touch controller. Other input devices 8072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein. The memory 809 may be used to store software programs as well as various data including, but not limited to, application programs and operating systems. The processor 810 may integrate an application processor, which mainly handles operating systems, user interfaces, application programs, etc., and a modem processor, which mainly handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into processor 810.
The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the above data quality monitoring method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.
The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and so on.
The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the data quality monitoring method embodiment, and can achieve the same technical effect, and the details are not repeated here to avoid repetition.
It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatus of the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present application may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present application.
While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A method for monitoring data quality, the method comprising:
determining a target check rule;
determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources;
according to the target verification rule and the target data source engine, verifying target data to determine a target verification result, wherein the target data come from the one or more target data sources;
and verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
2. The method of claim 1, prior to said determining a target verification rule, further comprising:
establishing a check rule database, wherein the check rule database is used for storing a plurality of rule configuration information and a plurality of engine configuration information;
the determining of the target verification rule comprises:
reading a data table in the check rule database to obtain target rule configuration information;
determining a target verification rule according to the target rule configuration information;
the target data source determining engine comprises:
reading a data table in the check rule database to obtain target engine configuration information;
and determining a target data source engine according to the target engine configuration information.
3. The method of claim 1, before said verifying the target data according to the target verification rule and determining the target verification result, further comprising:
and establishing corresponding engine links according to the target data source engine, and reading the target data from the one or more target data sources.
4. The method of claim 1, further comprising, after said obtaining the verification result:
and sending alarm information under the condition that the verification result is failed.
5. A data quality monitoring apparatus, the apparatus comprising:
the first determining module is used for determining a target verification rule;
a second determination module for determining a target data source engine, wherein the target data source engine is used for linking one or more target data sources;
the verification module is used for verifying target data according to the target verification rule and the target data source engine and determining a target verification result, wherein the target data come from the one or more target data sources;
and the verification module is used for verifying the target verification result according to a preset verification rule to obtain a verification result, wherein the preset verification rule comprises a preset verification value and a preset verification type.
6. The apparatus of claim 5, further comprising:
the system comprises a first establishing module, a second establishing module and a third establishing module, wherein the first establishing module is used for establishing a check rule database, and the check rule database is used for storing a plurality of rule configuration information and a plurality of engine configuration information;
the first determining module is configured to:
reading a data table in the check rule database to obtain target rule configuration information;
determining a target verification rule according to the target rule configuration information;
the second determining module is configured to:
reading a data table in the check rule database to obtain target engine configuration information;
and determining a target data source engine according to the target engine configuration information.
7. The apparatus of claim 5, further comprising:
and the second establishing module is used for establishing corresponding engine links according to the target data source engine and reading the target data from the one or more target data sources.
8. The apparatus of claim 5, further comprising:
and the alarm module is used for sending alarm information under the condition that the verification result is failed.
9. An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions when executed by the processor implementing the steps of the data quality monitoring method of any one of claims 1-4.
10. A readable storage medium, on which a program or instructions are stored, which program or instructions, when executed by a processor, carry out the steps of the data quality monitoring method according to any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210013830.2A CN114357032A (en) | 2022-01-06 | 2022-01-06 | Data quality monitoring method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210013830.2A CN114357032A (en) | 2022-01-06 | 2022-01-06 | Data quality monitoring method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114357032A true CN114357032A (en) | 2022-04-15 |
Family
ID=81106699
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210013830.2A Pending CN114357032A (en) | 2022-01-06 | 2022-01-06 | Data quality monitoring method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114357032A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115392811A (en) * | 2022-10-31 | 2022-11-25 | 成都飞机工业（集团）有限责任公司 | Method, device and equipment for evaluating quality of main data and storage medium |
-
2022
- 2022-01-06 CN CN202210013830.2A patent/CN114357032A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115392811A (en) * | 2022-10-31 | 2022-11-25 | 成都飞机工业（集团）有限责任公司 | Method, device and equipment for evaluating quality of main data and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10452465B2 (en) | Techniques for managing and analyzing log data | |
| CN111581291B (en) | Data processing method, device, electronic equipment and readable medium | |
| US10986012B1 (en) | System for generating alerts based on alert condition and optimistic concurrency control procedure | |
| CN111414407A (en) | Data query method and device of database, computer equipment and storage medium | |
| CN113268336B (en) | Service acquisition method, device, equipment and readable medium | |
| CN113489713A (en) | Network attack detection method, device, equipment and storage medium | |
| US20200250587A1 (en) | Framework for multi-tenant data science experiments at-scale | |
| CN112491602A (en) | Behavior data monitoring method and device, computer equipment and medium | |
| CN112000992B (en) | Data leakage prevention protection method and device, computer readable medium and electronic equipment | |
| CN111611276A (en) | Data query method, device and storage medium | |
| CN109542764B (en) | Webpage automatic testing method and device, computer equipment and storage medium | |
| CN114357032A (en) | Data quality monitoring method and device, electronic equipment and storage medium | |
| CN110879701B (en) | Workflow visual configuration method, server, system and medium | |
| US10725993B1 (en) | Indexing data sources using a highly available ETL for managed search | |
| CN115481026A (en) | Test case generation method and device, computer equipment and storage medium | |
| CA3144122A1 (en) | Data verifying method, device and system | |
| CN114281549A (en) | Data processing method and device | |
| CN113360916A (en) | Risk detection method, device, equipment and medium for application programming interface | |
| CN113722292B (en) | Disaster response processing method, device, equipment and storage medium of distributed data system | |
| US11379470B2 (en) | Techniques for concurrent data value commits | |
| CN110719260B (en) | Intelligent network security analysis method and device and computer readable storage medium | |
| CN114493536A (en) | New media operation information processing method, system, computer equipment and storage medium | |
| CN115309416A (en) | Data processing method, device and storage medium | |
| US10614090B1 (en) | Indexing and archiving streaming sources for managed search | |
| CN111193595A (en) | Error detection method, device, equipment and storage medium for electronic signature |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |