CN111352975A - Data quality management method, client, server and system - Google Patents
Data quality management method, client, server and system Download PDFInfo
- Publication number
- CN111352975A CN111352975A CN202010143464.3A CN202010143464A CN111352975A CN 111352975 A CN111352975 A CN 111352975A CN 202010143464 A CN202010143464 A CN 202010143464A CN 111352975 A CN111352975 A CN 111352975A
- Authority
- CN
- China
- Prior art keywords
- data
- data quality
- monitoring
- server
- result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007726 management method Methods 0.000 title claims abstract description 100
- 238000012544 monitoring process Methods 0.000 claims abstract description 191
- 230000002159 abnormal effect Effects 0.000 claims abstract description 42
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000004458 analytical method Methods 0.000 claims description 79
- 238000011156 evaluation Methods 0.000 claims description 58
- 238000013441 quality evaluation Methods 0.000 claims description 57
- 238000012550 audit Methods 0.000 claims description 31
- 230000003993 interaction Effects 0.000 claims description 25
- 238000007405 data analysis Methods 0.000 claims description 23
- 230000008569 process Effects 0.000 claims description 21
- 238000001514 detection method Methods 0.000 claims description 13
- 238000001303 quality assessment method Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000004806 packaging method and process Methods 0.000 claims description 8
- 238000003326 Quality management system Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 18
- 238000004891 communication Methods 0.000 description 7
- 238000012937 correction Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 238000013523 data management Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 239000002184 metal Substances 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000001376 precipitating effect Effects 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 238000004886 process control Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000012384 transportation and delivery Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2462—Approximate or statistical queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06395—Quality analysis or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/02—Banking, e.g. interest calculation or account maintenance
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Strategic Management (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- General Business, Economics & Management (AREA)
- Marketing (AREA)
- Finance (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Probability & Statistics with Applications (AREA)
- Educational Administration (AREA)
- Accounting & Taxation (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Technology Law (AREA)
- Game Theory and Decision Science (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Debugging And Monitoring (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a data quality management method, a client, a server and a system, and relates to the technical field of computers. One embodiment of the method comprises: acquiring a data quality requirement set by a first user, and sending the data quality requirement to a server, so that the server monitors stored data according to the data quality requirement; and receiving an abnormal monitoring result sent by the server. This embodiment enables rapid deployment of data quality monitoring.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a data quality management method, a client, a server and a system.
Background
Data quality requirements of data generated by commercial banks by regulatory agencies are increasingly pressing, and good data quality is a core target of data management and control.
The current way of monitoring data quality is mainly to monitor specific data according to a well-defined system. For example, some systems only monitor report results.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
at present, an effective method for implementing rapid deployment and monitoring on data quality problems is lacked.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data quality management method, a client, a server, and a system, which can quickly deploy data quality monitoring.
In order to achieve the above object, according to an aspect of the embodiments of the present invention, there is provided a data quality management method applied to a client, including:
acquiring a data quality requirement set by a first user, and sending the data quality requirement to a server, so that the server monitors stored data according to the data quality requirement;
and receiving an abnormal monitoring result sent by the server.
Preferably, after the acquiring the data quality requirement set by the first user, the method further includes:
providing the data quality requirement to a second user so that the second user can check the data quality requirement;
sending the result of the audit to the server side,
when the auditing result indicates that the auditing is passed, triggering the server to execute the step of monitoring the stored data according to the data quality requirement;
and when the result of the audit indicates that the audit does not pass, triggering the server to provide the audit opinion included in the result of the audit to the first user so as to enable the first user to modify the data quality requirement.
Preferably, the data quality management method further includes:
receiving a data analysis result sent by a server side, and providing the data analysis result to a third user, wherein the data analysis result comprises a plurality of analysis dimensions;
and acquiring at least one target analysis dimension selected by the third user, and sending the at least one target analysis dimension to the server, so that the server monitors data corresponding to the at least one target analysis dimension.
Preferably, the data quality management method further includes:
acquiring quality evaluation information input by a fourth user, and generating a corresponding quality evaluation request for the quality evaluation information;
sending the quality evaluation request to the server to enable the server to evaluate data/the abnormal monitoring result;
and receiving an evaluation result sent by the server side, and providing the evaluation result to the fourth user.
In a second aspect, a data quality management method applied to a server includes:
receiving a data quality requirement sent by a client, and monitoring stored data according to the data quality requirement;
and generating an abnormity monitoring result for the monitored abnormal data, and sending the abnormity monitoring result to the client.
Preferably, the first and second electrodes are formed of a metal,
the data quality management method further includes: constructing a plurality of monitoring tasks, wherein each monitoring task comprises a monitoring script and demand characteristic information;
the step of monitoring the stored data according to the data quality requirement comprises:
matching target demand characteristic information for the data quality demand;
and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data.
Preferably, the data quality management method further includes:
a plurality of data quality evaluation programs are set;
when a quality evaluation request is received, analyzing a feature identifier and an evaluation parameter of data to be evaluated, wherein the data to be evaluated is included in the quality evaluation request, and the evaluation parameter indicates a target to be evaluated and an evaluation standard;
packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated;
evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program;
and sending the evaluation result to the client.
Preferably, the data quality management method further includes:
and recording a plurality of data quality management nodes corresponding to the data quality management process, and monitoring and managing the process of each data quality management node.
Preferably, the data quality management method further includes:
carrying out data analysis on data in a data warehouse according to a preset data analysis strategy;
determining a parsing result, and retrieving whether there is a target parsing result satisfying a preset monitoring condition,
and if so, monitoring data corresponding to the target profiling result and providing an abnormal detection result.
Preferably, the data profiling policy includes:
any one or more of a frequency distribution analysis task, a data category analysis task, a column attribute analysis task, an integrity analysis task, an effectiveness analysis task, and a format analysis task.
In a third aspect, a client comprises: an acquisition unit and a first interaction unit, wherein,
the acquiring unit is used for acquiring the data quality requirement set by the first user;
the first interaction unit is configured to send the data quality requirement acquired by the acquisition unit to a server, so that the server monitors stored data according to the data quality requirement; and receiving an abnormal monitoring result sent by the server.
In a fourth aspect, a server includes: a second interaction unit and a monitoring unit, wherein,
the second interaction unit is used for receiving the data quality requirement sent by the client and sending the abnormal monitoring result monitored by the monitoring unit to the client;
and the monitoring unit is used for generating an abnormal monitoring result for the monitored abnormal data.
Preferably, the first and second electrodes are formed of a metal,
the monitoring unit is used for constructing a plurality of monitoring tasks, and each monitoring task comprises a monitoring script and demand characteristic information; matching target demand characteristic information for the data quality demand; and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data.
Preferably, the server further includes: a quality evaluation unit, wherein,
the second interaction unit is further configured to receive a quality assessment request sent by the client, and send a result of the quality assessment unit assessment to the client;
the quality evaluation unit is used for setting a plurality of data quality evaluation programs; analyzing the feature identifier of the data to be evaluated and the evaluation parameter included in the quality evaluation request received by the second interaction unit, wherein the evaluation parameter indicates the target to be evaluated and an evaluation standard; packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated; and evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program.
In a fifth aspect, a data quality management system includes: the client described in the above embodiment and the server described in the above embodiment.
One embodiment of the above invention has the following advantages or benefits: the data quality requirement set by the first user is obtained through the client, namely, the user only needs to deploy the data quality requirement needed by the user, the client interacts with the server, the server monitors the data related to the data quality requirement according to the data quality requirement, and the client receives an abnormal monitoring result sent by the server and can rapidly deploy data quality monitoring, so that the data quality monitoring deployment efficiency and the data quality monitoring efficiency are effectively improved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of the main flow of a method of data quality management according to an embodiment of the invention;
FIG. 2 is a schematic diagram of the main flow of a method of data quality management according to an embodiment of the invention;
FIG. 3 is a schematic diagram of the main flow of a method of data quality management according to an embodiment of the invention;
FIG. 4 is a schematic diagram of the main flow of a method of data quality management according to an embodiment of the invention;
FIG. 5 is a schematic structural diagram of data profiling according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of the main flow of a method of data quality management according to an embodiment of the invention;
FIG. 7 is a schematic diagram of the main elements of a client according to an embodiment of the invention;
FIG. 8 is a schematic diagram of the main elements of a server according to an embodiment of the invention;
FIG. 9 is a schematic diagram of the primary means of data quality management according to an embodiment of the present invention;
FIG. 10 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 11 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The requirements of the supervision organization on the quality of the supervision statistical data of the commercial bank are increasingly urgent, and in recent years, the supervision is gradually developed from the requirement of report results to the requirement of establishing a monitoring process from data acquisition to data application end-to-end. An effective data quality monitoring system is established in a commercial bank which requires daily monitoring in the 'good standards for managing bank supervision statistical data', the supervision statistical data quality is continuously monitored, powerful measures are taken, the responsibility of each link is implemented, and the data quality is ensured. Therefore, a unified data quality monitoring and management platform for banks is established, data quality problems are displayed in a centralized manner, root analysis is carried out on the quality problems, quality problem rectification is tracked and supervised, and a quantitative basis is provided for bank data quality management and assessment and evaluation. At present, domestic banks lack effective systems and tools to implement rapid deployment and monitoring on discovered data quality problems, and the system locates details of error data, supports root cause analysis and improves rectification efficiency.
Fig. 1 is a diagram of a data quality management method applied to a client according to an embodiment of the present invention. As shown in fig. 1, the data quality management method may include the steps of:
101: acquiring a data quality requirement set by a first user, and sending the data quality requirement to a server, so that the server monitors stored data according to the data quality requirement;
102: and receiving an abnormal monitoring result sent by the server.
In the embodiment shown in fig. 1, the data quality requirement set by the first user is acquired through the client, that is, the user only needs to deploy the data quality requirement needed by the user, the server performs monitoring related to the data quality requirement according to the data quality requirement through interaction between the client and the server, and the client receives an abnormal monitoring result sent by the server, so that data quality monitoring can be rapidly deployed, and thus, the data quality monitoring deployment efficiency and the data quality monitoring efficiency are effectively improved.
In an embodiment of the present invention, as shown in fig. 2, after the step 101, the following steps may be further included:
201: providing the data quality requirement to a second user so that the second user can check the data quality requirement;
202: sending the result of the audit to the server, and executing step 203 when the result of the audit indicates that the audit passes; when the result of the audit indicates that the audit does not pass, executing step 204;
203: triggering the server to execute the step of monitoring the stored data according to the data quality requirement, and ending the current process;
204: and triggering the server to provide the auditing opinions included by the auditing results to the first user so as to enable the first user to modify the data quality requirement.
The specific implementation manner of providing the data quality requirement to the second user in step 201 may be that the client of the first user directly sends the data quality requirement to the client of the second user, or that the client of the first user sends the data quality requirement to the server, and the server serves as a relay and transfers the data quality requirement to the client of the second user.
On one hand, the client side for realizing the data monitoring method can distribute different authorities to different users, for example, a first user has a setting authority for setting data quality requirements, and a second user has an auditing authority for auditing the data quality requirements, so that the data security is ensured.
In addition, the standardization of data quality requirements is guaranteed through the process, so that the data quality requirements can be accurately identified by the server, and the execution accuracy of the monitoring task of the server is guaranteed.
In an embodiment of the present invention, the data quality management method may further include: receiving a data analysis result sent by a server side, and providing the data analysis result to a third user, wherein the data analysis result comprises a plurality of analysis dimensions; and acquiring at least one target analysis dimension selected by the third user, and sending the at least one target analysis dimension to the server, so that the server monitors data corresponding to the at least one target analysis dimension. The data monitoring method and the data monitoring device realize that corresponding data monitoring is set according to the result of data analysis so as to realize monitoring of the data quality and monitoring according to the actual requirement of the data.
In an embodiment of the present invention, the data quality management method may further include: acquiring quality evaluation information input by a fourth user, and generating a corresponding quality evaluation request for the quality evaluation information; sending the quality evaluation request to the server to enable the server to evaluate data/the abnormal monitoring result; and receiving an evaluation result sent by the server side, and providing the evaluation result to the fourth user.
Through the evaluation, on one hand, the accuracy of the monitoring result of the data quality management method can be evaluated, and on the other hand, the stored data of the bank can be evaluated so as to evaluate the control of the bank on the data quality and further monitor the data quality, thereby further ensuring the data quality.
Fig. 3 is a diagram illustrating a data quality management method according to an embodiment of the present invention, applied to a server. As shown in fig. 3, the data quality management method may include the steps of:
301: receiving a data quality requirement sent by a client, and monitoring stored data according to the data quality requirement;
302: and generating an abnormity monitoring result for the monitored abnormal data, and sending the abnormity monitoring result to the client.
In an embodiment of the present invention, the data quality management method may further include: constructing a plurality of monitoring tasks, wherein each monitoring task comprises a monitoring script and demand characteristic information; accordingly, the step of monitoring the stored data according to the data quality requirement comprises: matching target demand characteristic information for the data quality demand; and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data. The monitoring tasks are flexibly configured according to the data quality requirements, and therefore data quality management can be rapidly achieved.
In an embodiment of the present invention, as shown in fig. 4, the data quality management method may further include the following steps to achieve data quality evaluation:
401: the server is provided with a plurality of data quality evaluation programs;
402: when a quality evaluation request is received, analyzing a feature identifier and an evaluation parameter of data to be evaluated, wherein the data to be evaluated is included in the quality evaluation request, and the evaluation parameter indicates a target to be evaluated and an evaluation standard;
403: packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated;
404: evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program;
405: and sending the evaluation result to the client.
It should be noted that the data quality evaluation process may include evaluating the data quality of the entire row level and evaluating the data quality managed by the data quality management method provided in the embodiment of the present invention. The evaluation of the data quality of the whole line level is implemented by an information technology department, and the evaluation of the data quality of the information system level is implemented by an information system service administration department. The information technology department can make a full-line data quality evaluation scheme through data quality requirements according to the supervision requirements of a bank prisoner and a people bank and full-line data quality management targets through the client, and evaluate the data quality level of each business department and each branch line. And the data management and control department feeds the evaluation result of the all-data quality back to the service administration department of the relevant information system, and supervises the data management and control department to correct and improve the data quality problem.
In an embodiment of the present invention, the data quality management method may further include: and recording a plurality of data quality management nodes corresponding to the data quality management process, and monitoring and managing the process of each data quality management node.
In the embodiment of the invention, the data quality requirement management refers to a working process of overall management aiming at various requirements such as data quality monitoring requirements, data quality problem rectification requirements, data quality evaluation requirements and the like related to data quality. The data quality requirements come from the requirements of external supervision, business departments, information technology departments and the like for solving the data quality condition of bank data, solving the data quality problem and improving the data quality. Such as data quality evaluation of downstream system supply by business department; the information technology department corrects the problem of the data quality of the whole row level; the evaluation and the modification are based on the result of data quality monitoring, and a certain gate can also directly carry out the quality monitoring requirement of a certain rule on the data object.
The plurality of data quality management nodes may include: data quality requirements, demand assessment, data quality monitoring, quality problem correction, monitoring assessment, quality assessment, data quality problem correction, and the like, which may be monitored to track the state of data quality management.
In an embodiment of the present invention, the data quality management method may further include: carrying out data analysis on data in a data warehouse according to a preset data analysis strategy; determining a parsing result, searching whether an object parsing result meeting preset monitoring conditions exists, if so, monitoring data corresponding to the object parsing result, and providing an abnormal detection result.
Data profiling data in a data warehouse includes: and comprehensively scanning the record value condition of the data warehouse field according to a preset general inspection rule, and discovering potential data quality problems through abnormal value analysis. The result of data analysis is not the data quality problem, and the data analysis is to make a comprehensive analysis and display of the current situation of the data, can help data quality managers to find potential problems, and can be used as the input of the data quality requirement.
Accordingly, in one embodiment of the present invention, the data parsing strategy may include: any one or more of a frequency distribution analysis task, a data category analysis task, a column attribute analysis task, an integrity analysis task, an effectiveness analysis task, and a format analysis task.
Specific contents of data analysis (a frequency distribution analysis task, a data category analysis task, a column attribute analysis task, an integrity analysis task, an effectiveness analysis task, and a format analysis task) that can be realized by the embodiment of the present invention will be analyzed in detail below:
as shown in fig. 5, the frequency distribution is to perform deduplication on all record values of the analysis object (field), and count the frequency distribution condition (i.e., the cumulative occurrence number of record values, the frequency distribution of record values, and the frequency distribution percentage of record values) of each value of the deduplicated record values, the data type of the record values, the length of the record values, and the format of the record values, and is the basis of the following four analysis operations.
The data category analysis is to judge the data category of the field according to the characteristics of the actual recorded value after the duplication removal, and the main data categories comprise a coding category, a code category, an indicator category, a date category (including a time category and a date-time category), a numerical value category (including an amount category and a percentage category), a text category and an unknown category. The emphasis of data analysis of different classes is different, so that the data class analysis is the basis for organizing and focusing subsequent three analysis works.
The attribute analysis is to evaluate the accuracy of the metadata definition of the field according to the de-duplicated actual recorded value, and the specific analysis content includes data type, length (value class), precision (value class), nullability and uniqueness. Fields of different data classes are of different applicability in this analysis, e.g., length (value class), precision (value class) analysis apply only fields whose data class is "value class".
The integrity and validity analysis is to judge the integrity and validity of each record value according to the de-duplicated actual record values according to a preset rule, and is a basic evaluation analysis of data quality. The validity analysis is only for recorded values that meet the integrity requirement, so the integrity analysis needs to be completed first, and then the validity analysis needs to be performed.
The format analysis is to convert the de-duplicated actual record value into a general format according to a preset conversion rule and count the accumulated occurrence times of the converted general format. The format analysis pair can be used to find potential data quality problems with format requirements or coding requirements. The different data types affect the importance of format analysis, and the fields with the data types of "encoding type" and "date type" are required to be subjected to format analysis, but the analysis has not so high importance for the fields with the data types of "text type".
In an embodiment of the present invention, the data profiling may include creating a profiling item, performing data profiling, and deriving a profiling report, wherein the creating of the profiling item, the performing of the data profiling, and the deriving of the profiling report may be automatically performed according to a setting of a user.
In addition, in an embodiment of the present invention, the data quality management method may further include: a data quality knowledge base is provided which can be used to store various rules, institutional regulations, rectification schemes, etc.
The data quality knowledge base is an empirical summary of data quality management, and comprises an empirical summary of institutional regulations related to data quality, data quality process control rules (which can be from implementation process requirement delivery parts), phenomena, reasons, processing methods and the like of common data quality problems. The knowledge can be used as a reference for solving the data quality problem in the future, and is indexed and classified and managed in a keyword mode.
The data quality knowledge base mainly comprises a conventional knowledge base and a data quality knowledge base.
Conventional knowledge bases include, but are not limited to: data quality related supervision policies, full-line system, management method, operation procedures, evaluation scheme/model description, data quality rule table of each application component, and the like.
The data quality knowledge base generally consists of the following components:
(1) data quality monitoring rules: the inspection rule is a precondition for finding data quality problems, is a measure of data quality, and becomes a part of data quality knowledge.
(2) Data quality issues and root causes: data quality issues are an important component of data quality knowledge. The data quality problem content comprises problem description, problem severity, objects related to the problems, problem responsible persons, root cause description and the like.
(3) The rectification scheme is as follows: the correction scheme is the core of data quality knowledge, the correction scheme is a principle, a processing method and a specific measure which are provided aiming at the data quality problem, and the composition content of the correction scheme can be a text, and can also be a picture, a table or a file attachment.
In addition, the data quality management team should choose to include knowledge that data quality issues have been closed or conditionally closed in the knowledge base.
In an embodiment of the present invention, the data management method may further include: and generating a data quality report for the data monitoring result.
The data quality report includes two categories of data quality management statistics report and data quality report.
The data quality statistical report is a statistical analysis report of daily working conditions of data quality, and the specific content is analyzed and displayed according to the characteristics and requirements of data quality demand management, data quality monitoring, data quality problem management and data quality evaluation of each block.
The data quality report is a carrier for reporting the overall data quality condition to the leader of the management layer and related departments periodically, and comprises information such as current conditions, change trends, key problems, related influences, follow-up measures and the like, so that the leader of the decision layer can know the overall data quality management and main problems, make key decisions and guide and promote the data quality management work.
The data quality management method provided by the embodiment of the invention promotes the correction of the tracking data quality problem, quantitatively evaluates the quality level of the whole row of data, standardizes the quality management process of the whole row of data and realizes the centralized professional management of the whole row of data quality.
The data quality management method provided by the embodiment of the invention realizes the centralized monitoring of the quality of all-row data, and realizes the centralized monitoring of the quality of all-row data and the centralized management monitoring requirement by taking a data warehouse as a basis; parameterizing and flexibly configuring a monitoring rule, realizing rapid deployment of a data quality monitoring task, and responding to a service requirement in time; and realizing the visual display of the monitoring result.
In addition, the scheme provided by the embodiment of the invention supports supervision and urging of data quality problem rectification through a data quality evaluation process, and realizes data quality problem discovery, root cause analysis and problem rectification whole process management by recording each node in the data quality management; centralized management and classification of data quality problems are realized; root cause analysis and responsibility positioning of data quality problems are supported; supporting the design of a data quality problem rectification scheme; and the rectification tracking and the rectification effect verification are realized.
In addition, the scheme provided by the embodiment of the invention quantitatively evaluates the data quality through a data quality evaluation process, supports evaluation and realizes analysis of multi-dimensional data quality conditions and change trends; and a quantifiable data base is provided for the implementation of data quality management and assessment of all business departments.
The data managed by the data quality management method can be channel data, customer information, data in a bank data warehouse, personal customer data, public customer data, employee data and the like.
In summary, the data quality management method provided by the embodiment of the invention mainly includes data demand management, data quality monitoring, problem rectification, quality evaluation, a knowledge base and the like. The data quality management is a unified entry of data quality management requirements, and the data quality requirements proposed by all departments are managed in a centralized manner, so that the identification and centralized management of data quality problems are realized; the data quality monitoring is to convert the problem to be monitored into a script executable by a data warehouse by using a rule template through a natural language, and obtain a monitoring result and error details from the data warehouse; problem rectification refers to a process of analyzing root cause aiming at data quality problems found in monitoring or evaluation, designing a rectification scheme, decomposing into rectification tasks, conducting rectification by responsible parties, and verifying rectification effect in a system; the knowledge base is based on the problem of data quality and the different rectification schemes, and is designed for precipitating the prior management experience. Therefore, effective identification of the quality problem of the whole line of data is realized, and centralized monitoring and rectification tracking are realized, so that the quality of the whole line of data is continuously improved, and the user experience is improved.
The following describes the data quality management method in detail by taking the interaction among the client, the server and the data warehouse as an example to perform quality management on the business data of the bank stored in the data warehouse. As shown in fig. 6, the data quality management method may include the steps of:
600: the server side constructs a plurality of monitoring tasks;
each monitoring task comprises a monitoring script and demand characteristic information;
601: the client acquires the data quality requirement set by a first user;
602: the client provides the data quality requirement for the second user, so that the second user can audit the data quality requirement, and when the audit result is that the data quality requirement does not pass, step 603 is executed; when the auditing result is passed, executing step 604;
603: the client provides the adjustment and modification requirement of the second user to the first user so that the user can adjust and modify the data quality requirement;
604: the client sends the data quality requirement to the server;
605: the server matches target demand characteristic information for the data quality demand;
606: the server side determines a target monitoring task according to the target demand characteristic information;
607: the server side provides a monitoring script included by the target monitoring task to a data warehouse;
608: the data warehouse runs a monitoring script included in the target monitoring task and monitors stored data;
609: the data warehouse sends the monitoring result to the server;
610: the server side generates a monitoring report for a monitoring result;
611: the server side sends the monitoring report to the client side;
612: the client acquires quality evaluation information input by a fourth user, and generates a corresponding quality evaluation request for the quality evaluation information;
613: the client sends the quality evaluation request to the server;
614: the server evaluates the data/abnormal monitoring result;
the specific implementation manner of the step is as follows: the server is provided with a plurality of data quality evaluation programs; when a quality evaluation request is received, analyzing a feature identifier and an evaluation parameter of data to be evaluated, wherein the data to be evaluated is included in the quality evaluation request, and the evaluation parameter indicates a target to be evaluated and an evaluation standard; packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated; and evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program.
615: the client receives an evaluation result sent by the server and provides the evaluation result to the fourth user;
616: the server analyzes the data in the data warehouse according to a preset data analyzing strategy;
617: the server determines the parsing result, and retrieves whether there is a target parsing result meeting a preset monitoring condition, if yes, step 621 is executed; otherwise, go to step 618;
the monitoring condition may be that an error rate of a certain item in the parsing result reaches a certain threshold, and the item is the target parsing result.
618: the server side sends the analysis result to the client side;
619: the client selects a target analysis result from the analysis results;
620: the client sends the target analysis result to the server;
621: and the server monitors data corresponding to the target analysis result and provides an abnormal detection result.
In addition, the data quality management method provided by the embodiment of the invention can be developed based on an agile framework, thereby effectively reducing the development cost.
As shown in fig. 7, an embodiment of the present invention provides a client 700, where the client 700 may include: an acquisition unit 701 and a first interaction unit 702, wherein,
the obtaining unit 701 is configured to obtain a data quality requirement set by a first user;
the first interaction unit 702 is configured to send the data quality requirement acquired by the acquisition unit 701 to a server, so that the server monitors stored data according to the data quality requirement; and receiving an abnormal monitoring result sent by the server.
In an embodiment of the present invention, the client 700 may further include: a unit (not shown in the figures) is provided, in which,
a providing unit (not shown in the figures) for providing said data quality requirement to a second user;
the first interaction unit 702 is further configured to send an audit result to the server, and when the audit result indicates that the audit is passed, trigger the server to perform the step of monitoring the stored data according to the data quality requirement; and when the result of the audit indicates that the audit does not pass, triggering the server to provide the audit opinion included in the result of the audit to the first user so as to enable the first user to modify the data quality requirement.
In an embodiment of the present invention, the first interaction unit 702 is further configured to receive a result of data parsing sent by a server, and provide the result of data parsing to a third user, where the result of data parsing includes a plurality of parsing dimensions; sending the at least one target analysis dimension to the server, so that the server monitors data corresponding to the at least one target analysis dimension;
the obtaining unit 701 is further configured to obtain at least one target parsing dimension selected by the third user.
In an embodiment of the present invention, the obtaining unit 701 is further configured to obtain quality assessment information input by a fourth user, and generate a corresponding quality assessment request for the quality assessment information;
the first interaction unit 702 is further configured to send the quality evaluation request to the server, so that the server evaluates data/the anomaly monitoring result; and receiving an evaluation result sent by the server side, and providing the evaluation result to the fourth user.
As shown in fig. 8, an embodiment of the present invention provides a server 800, where the server 800 may include: a second interaction unit 801 and a monitoring unit 802, wherein,
the second interaction unit 801 is configured to receive a data quality requirement sent by a client, and send an abnormal monitoring result monitored by the monitoring unit 802 to the client;
the monitoring unit 802 is configured to generate an anomaly monitoring result for the monitored anomaly data.
In an embodiment of the present invention, the monitoring unit 802 is configured to construct a plurality of monitoring tasks, where each of the monitoring tasks includes a monitoring script and requirement characteristic information; matching target demand characteristic information for the data quality demand; and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data.
In an embodiment of the present invention, the server 800 may further include: a quality evaluation unit (not shown in the figure) in which,
the second interaction unit 801 is further configured to receive a quality assessment request sent by the client, and send a result of the quality assessment unit assessment to the client;
the quality evaluation unit (not shown in the figure) is used for setting a plurality of data quality evaluation programs; analyzing the feature identifier of the data to be evaluated and the evaluation parameter included in the quality evaluation request received by the second interaction unit, wherein the evaluation parameter indicates the target to be evaluated and an evaluation standard; packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated; and evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program.
In an embodiment of the present invention, the monitoring unit 802 is further configured to record a plurality of data quality management nodes corresponding to a data quality management process, and monitor and manage a process of each of the data quality management nodes.
In an embodiment of the present invention, the monitoring unit 802 is further configured to perform data parsing on data in a data warehouse according to a preset data parsing policy; determining a parsing result, searching whether an object parsing result meeting preset monitoring conditions exists, if so, monitoring data corresponding to the object parsing result, and providing an abnormal detection result.
In one embodiment of the present invention, the data parsing policy includes: any one or more of a frequency distribution analysis task, a data category analysis task, a column attribute analysis task, an integrity analysis task, an effectiveness analysis task, and a format analysis task.
As shown in fig. 9, an embodiment of the present invention provides a data quality management system 900, where the data quality management system 900 includes: the client 700 provided in any of the above embodiments and the server 800 provided in any of the above embodiments.
Fig. 10 shows an exemplary system architecture 1000 of a data quality management method or a data quality management client or a data quality management server to which embodiments of the present invention may be applied.
As shown in fig. 10, the system architecture 1000 may include terminal devices 1001, 1002, 1003, a network 1004, and a server 1005. The network 1004 is used to provide a medium for communication links between the terminal devices 1001, 1002, 1003 and the server 1005. Network 1004 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 1001, 1002, 1003 to interact with a server 1005 via a network 1004 to receive or transmit messages or the like. The terminal devices 1001, 1002, and 1003 may have installed thereon a client provided by the embodiment of the present invention and various communication client applications, such as a web browser application, a search application, an instant messaging tool, a mailbox client, social platform software, and the like (for example only).
The terminal devices 1001, 1002, 1003 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 1005 may be a server providing various services, such as a backend management server (for example only) providing support for data quality requirements sent by users with the terminal devices 1001, 1002, 1003. The background management server may analyze the received data such as the data quality requirement, allocate a corresponding monitoring task to the data quality requirement, and feed back a processing result (for example, data quality is abnormal — only an example) to the terminal device.
It should be noted that the data quality management method provided by the embodiment of the present invention is generally executed by the server 1005 and the terminal devices 1001, 1002, and 1003 in combination, and accordingly, the data quality management server is generally disposed in the server 1005, and the data quality management client is generally disposed in the terminal devices 1001, 1002, and 1003.
It should be understood that the number of terminal devices, networks, and servers in fig. 10 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 11, shown is a block diagram of a computer system 1100 suitable for use with a terminal device or server implementing an embodiment of the present invention. The terminal device shown in fig. 11 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 11, the computer system 1100 includes a Central Processing Unit (CPU)1101, which can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)1102 or a program loaded from a storage section 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data necessary for the operation of the system 1100 are also stored. The CPU 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.
The following components are connected to the I/O interface 1105: an input portion 1106 including a keyboard, mouse, and the like; an output portion 1107 including a signal output unit such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and a speaker; a storage section 1108 including a hard disk and the like; and a communication section 1109 including a network interface card such as a LAN card, a modem, or the like. The communication section 1109 performs communication processing via a network such as the internet. A driver 1110 is also connected to the I/O interface 1105 as necessary. A removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1110 as necessary, so that a computer program read out therefrom is mounted into the storage section 1108 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication portion 1109 and/or installed from the removable medium 1111. The above-described functions defined in the system of the present invention are executed when the computer program is executed by a Central Processing Unit (CPU) 1101.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present invention may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes a second interaction unit and a monitoring unit. The names of these units do not form a limitation to the unit itself in some cases, for example, the second interactive unit may also be described as "a unit that receives data quality requirements sent by the client and sends the anomaly monitoring result to the client.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring a data quality requirement set by a first user, and sending the data quality requirement to a server, so that the server monitors stored data according to the data quality requirement; and receiving an abnormal monitoring result sent by the server.
The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: providing the data quality requirement to a second user so that the second user can check the data quality requirement; sending the result of the audit to the server, and triggering the server to execute the step of monitoring the stored data according to the data quality requirement when the result of the audit indicates that the audit is passed; and when the result of the audit indicates that the audit does not pass, triggering the server to provide the audit opinion included in the result of the audit to the first user so as to enable the first user to modify the data quality requirement.
The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a data analysis result sent by a server side, and providing the data analysis result to a third user, wherein the data analysis result comprises a plurality of analysis dimensions; and acquiring at least one target analysis dimension selected by the third user, and sending the at least one target analysis dimension to the server, so that the server monitors data corresponding to the at least one target analysis dimension.
The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: acquiring quality evaluation information input by a fourth user, and generating a corresponding quality evaluation request for the quality evaluation information; sending the quality evaluation request to the server to enable the server to evaluate data/the abnormal monitoring result; and receiving an evaluation result sent by the server side, and providing the evaluation result to the fourth user.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: receiving a data quality requirement sent by a client, and monitoring stored data according to the data quality requirement; and generating an abnormity monitoring result for the monitored abnormal data, and sending the abnormity monitoring result to the client.
The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: constructing a plurality of monitoring tasks, wherein each monitoring task comprises a monitoring script and demand characteristic information; the step of monitoring the stored data according to the data quality requirement comprises: matching target demand characteristic information for the data quality demand; and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data.
The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise: a plurality of data quality evaluation programs are set; when a quality evaluation request is received, analyzing a feature identifier and an evaluation parameter of data to be evaluated, wherein the data to be evaluated is included in the quality evaluation request, and the evaluation parameter indicates a target to be evaluated and an evaluation standard; packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated; evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program; and sending the evaluation result to the client.
According to the technical scheme of the embodiment of the invention, the data quality requirement set by the first user is obtained through the client, namely, the user only needs to deploy the data quality requirement needed by the user, the server carries out monitoring related to the data quality requirement according to the data quality requirement through interaction between the client and the server, and the client receives an abnormal monitoring result sent by the server, so that data quality monitoring can be rapidly deployed, the data quality monitoring deployment efficiency and the data quality monitoring efficiency are effectively improved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (17)
1. A data quality management method is applied to a client and comprises the following steps:
acquiring a data quality requirement set by a first user, and sending the data quality requirement to a server, so that the server monitors stored data according to the data quality requirement;
and receiving an abnormal monitoring result sent by the server.
2. The data quality management method according to claim 1, further comprising, after said obtaining the data quality requirement set by the first user:
providing the data quality requirement to a second user so that the second user can check the data quality requirement;
sending the result of the audit to the server side,
when the auditing result indicates that the auditing is passed, triggering the server to execute the step of monitoring the stored data according to the data quality requirement;
and when the result of the audit indicates that the audit does not pass, triggering the server to provide the audit opinion included in the result of the audit to the first user so as to enable the first user to modify the data quality requirement.
3. The data quality management method of claim 1, further comprising:
receiving a data analysis result sent by a server side, and providing the data analysis result to a third user, wherein the data analysis result comprises a plurality of analysis dimensions;
and acquiring at least one target analysis dimension selected by the third user, and sending the at least one target analysis dimension to the server, so that the server monitors data corresponding to the at least one target analysis dimension.
4. The data quality management method according to any one of claims 1 to 3, characterized by further comprising:
acquiring quality evaluation information input by a fourth user, and generating a corresponding quality evaluation request for the quality evaluation information;
sending the quality evaluation request to the server to enable the server to evaluate data/the abnormal monitoring result;
and receiving an evaluation result sent by the server side, and providing the evaluation result to the fourth user.
5. A data quality management method is applied to a server and comprises the following steps:
receiving a data quality requirement sent by a client, and monitoring stored data according to the data quality requirement;
and generating an abnormity monitoring result for the monitored abnormal data, and sending the abnormity monitoring result to the client.
6. The data quality management method according to claim 5,
further comprising: constructing a plurality of monitoring tasks, wherein each monitoring task comprises a monitoring script and demand characteristic information;
the step of monitoring the stored data according to the data quality requirement comprises:
matching target demand characteristic information for the data quality demand;
and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data.
7. The data quality management method of claim 5, further comprising:
a plurality of data quality evaluation programs are set;
when a quality evaluation request is received, analyzing a feature identifier and an evaluation parameter of data to be evaluated, wherein the data to be evaluated is included in the quality evaluation request, and the evaluation parameter indicates a target to be evaluated and an evaluation standard;
packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated;
evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program;
and sending the evaluation result to the client.
8. The data quality management method according to any one of claims 5 to 7, characterized by further comprising:
and recording a plurality of data quality management nodes corresponding to the data quality management process, and monitoring and managing the process of each data quality management node.
9. The data quality management method according to any one of claims 5 to 7, characterized by further comprising:
carrying out data analysis on data in a data warehouse according to a preset data analysis strategy;
determining a parsing result, and retrieving whether there is a target parsing result satisfying a preset monitoring condition,
and if so, monitoring data corresponding to the target profiling result and providing an abnormal detection result.
10. The data quality management method of claim 9, wherein the data profiling policy comprises:
any one or more of a frequency distribution analysis task, a data category analysis task, a column attribute analysis task, an integrity analysis task, an effectiveness analysis task, and a format analysis task.
11. A client, comprising: an acquisition unit and a first interaction unit, wherein,
the acquiring unit is used for acquiring the data quality requirement set by the first user;
the first interaction unit is configured to send the data quality requirement acquired by the acquisition unit to a server, so that the server monitors stored data according to the data quality requirement; and receiving an abnormal monitoring result sent by the server.
12. A server, comprising: a second interaction unit and a monitoring unit, wherein,
the second interaction unit is used for receiving the data quality requirement sent by the client and sending the abnormal monitoring result monitored by the monitoring unit to the client;
and the monitoring unit is used for generating an abnormal monitoring result for the monitored abnormal data.
13. The server according to claim 12,
the monitoring unit is used for constructing a plurality of monitoring tasks, and each monitoring task comprises a monitoring script and demand characteristic information; matching target demand characteristic information for the data quality demand; and determining a target monitoring task according to the target demand characteristic information, and providing a monitoring script included by the target monitoring task to a data warehouse so that the data warehouse runs the monitoring script included by the target monitoring task to monitor the stored data.
14. The server according to claim 12, further comprising: a quality evaluation unit, wherein,
the second interaction unit is further configured to receive a quality assessment request sent by the client, and send a result of the quality assessment unit assessment to the client;
the quality evaluation unit is used for setting a plurality of data quality evaluation programs; analyzing the feature identifier of the data to be evaluated and the evaluation parameter included in the quality evaluation request received by the second interaction unit, wherein the evaluation parameter indicates the target to be evaluated and an evaluation standard; packaging the target to be evaluated and the evaluation standard to a data quality evaluation program corresponding to the feature identifier of the data to be evaluated; and evaluating the data/abnormal detection result corresponding to the target to be evaluated stored in the data warehouse by using the packaged data quality evaluation program.
15. A data quality management system, comprising: the client of claim 11 and the server of any of claims 12 to 14.
16. An electronic device for data quality management, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-10.
17. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010143464.3A CN111352975B (en) | 2020-03-04 | 2020-03-04 | Data quality management method, client, server and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010143464.3A CN111352975B (en) | 2020-03-04 | 2020-03-04 | Data quality management method, client, server and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111352975A true CN111352975A (en) | 2020-06-30 |
| CN111352975B CN111352975B (en) | 2024-01-30 |
Family
ID=71196102
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010143464.3A Active CN111352975B (en) | 2020-03-04 | 2020-03-04 | Data quality management method, client, server and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111352975B (en) |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012158649A2 (en) * | 2011-05-14 | 2012-11-22 | Luca Anastasia Maria | System and method for objective assessment of learning outcomes |
| CN102916839A (en) * | 2012-10-26 | 2013-02-06 | 南宁职业技术学院 | Automatic monitoring system for agricultural work in sugarhouse |
| CN104683473A (en) * | 2015-03-13 | 2015-06-03 | 百度在线网络技术（北京）有限公司 | Service quality monitoring method, server side, client and system |
| CN105071989A (en) * | 2015-07-30 | 2015-11-18 | 世纪龙信息网络有限责任公司 | Video content distribution quality monitoring system and monitoring method therefor |
| WO2018103521A1 (en) * | 2016-12-08 | 2018-06-14 | 腾讯科技（深圳）有限公司 | Monitoring method for server, device, and storage medium |
| CN108520340A (en) * | 2018-03-22 | 2018-09-11 | 湖南润安危物联科技发展有限公司 | A kind of data processing method, supervisory systems and storage medium |
| CN109445922A (en) * | 2018-10-31 | 2019-03-08 | 北京慧流科技有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN109522287A (en) * | 2018-09-18 | 2019-03-26 | 平安科技（深圳）有限公司 | Monitoring method, system, equipment and the medium of distributed document storage cluster |
| CN109784736A (en) * | 2019-01-21 | 2019-05-21 | 成都乐超人科技有限公司 | A kind of analysis and decision system based on big data |
| CN109857657A (en) * | 2019-01-18 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Code detection method, device, computer equipment and storage medium |
| CN110377569A (en) * | 2019-06-19 | 2019-10-25 | 中国平安人寿保险股份有限公司 | Log monitoring method, device, computer equipment and storage medium |
| CN110445637A (en) * | 2019-07-05 | 2019-11-12 | 深圳壹账通智能科技有限公司 | Event-monitoring method, system, computer equipment and storage medium |
-
2020
- 2020-03-04 CN CN202010143464.3A patent/CN111352975B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012158649A2 (en) * | 2011-05-14 | 2012-11-22 | Luca Anastasia Maria | System and method for objective assessment of learning outcomes |
| CN102916839A (en) * | 2012-10-26 | 2013-02-06 | 南宁职业技术学院 | Automatic monitoring system for agricultural work in sugarhouse |
| CN104683473A (en) * | 2015-03-13 | 2015-06-03 | 百度在线网络技术（北京）有限公司 | Service quality monitoring method, server side, client and system |
| CN105071989A (en) * | 2015-07-30 | 2015-11-18 | 世纪龙信息网络有限责任公司 | Video content distribution quality monitoring system and monitoring method therefor |
| WO2018103521A1 (en) * | 2016-12-08 | 2018-06-14 | 腾讯科技（深圳）有限公司 | Monitoring method for server, device, and storage medium |
| CN108520340A (en) * | 2018-03-22 | 2018-09-11 | 湖南润安危物联科技发展有限公司 | A kind of data processing method, supervisory systems and storage medium |
| CN109522287A (en) * | 2018-09-18 | 2019-03-26 | 平安科技（深圳）有限公司 | Monitoring method, system, equipment and the medium of distributed document storage cluster |
| CN109445922A (en) * | 2018-10-31 | 2019-03-08 | 北京慧流科技有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN109857657A (en) * | 2019-01-18 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Code detection method, device, computer equipment and storage medium |
| CN109784736A (en) * | 2019-01-21 | 2019-05-21 | 成都乐超人科技有限公司 | A kind of analysis and decision system based on big data |
| CN110377569A (en) * | 2019-06-19 | 2019-10-25 | 中国平安人寿保险股份有限公司 | Log monitoring method, device, computer equipment and storage medium |
| CN110445637A (en) * | 2019-07-05 | 2019-11-12 | 深圳壹账通智能科技有限公司 | Event-monitoring method, system, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111352975B (en) | 2024-01-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11182394B2 (en) | Performing database file management using statistics maintenance and column similarity | |
| US8060532B2 (en) | Determining suitability of entity to provide products or services based on factors of acquisition context | |
| US9336259B1 (en) | Method and apparatus for historical analysis analytics | |
| CN109344170B (en) | Stream data processing method, system, electronic device and readable storage medium | |
| US20170192872A1 (en) | Interactive detection of system anomalies | |
| US20180046956A1 (en) | Warning About Steps That Lead to an Unsuccessful Execution of a Business Process | |
| US11144582B2 (en) | Method and system for parsing and aggregating unstructured data objects | |
| US20170109667A1 (en) | Automaton-Based Identification of Executions of a Business Process | |
| WO2019041925A1 (en) | Workflow data processing method and device, storage medium, and computer apparatus | |
| CN109120428B (en) | Method and system for wind control analysis | |
| US20170109639A1 (en) | General Model for Linking Between Nonconsecutively Performed Steps in Business Processes | |
| US20210286885A1 (en) | Method and system for enhancing data privacy of an industrial system or electric power system | |
| US20190179927A1 (en) | Enterprise data services cockpit | |
| US10073726B2 (en) | Detection of outage in cloud based service using usage data based error signals | |
| US20200097579A1 (en) | Detecting anomalous transactions in computer log files | |
| US20190354991A1 (en) | System and method for managing service requests | |
| US8543552B2 (en) | Detecting statistical variation from unclassified process log | |
| US9444708B2 (en) | Detection of outage in cloud based service using synthetic measurements and anonymized usage data | |
| US20220292006A1 (en) | System for Automatically Generating Insights by Analysing Telemetric Data | |
| US20170109640A1 (en) | Generation of Candidate Sequences Using Crowd-Based Seeds of Commonly-Performed Steps of a Business Process | |
| CN113836237A (en) | Method and device for auditing data operation of database | |
| US20170109637A1 (en) | Crowd-Based Model for Identifying Nonconsecutive Executions of a Business Process | |
| US9659266B2 (en) | Enterprise intelligence (‘EI’) management in an EI framework | |
| US20180139220A1 (en) | Shared capability system | |
| US20170109670A1 (en) | Crowd-Based Patterns for Identifying Executions of Business Processes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220927 Address after: 12 / F, 15 / F, 99 Yincheng Road, China (Shanghai) pilot Free Trade Zone, Pudong New Area, Shanghai, 200120 Applicant after: Jianxin Financial Science and Technology Co.,Ltd. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |