CN113342791A - Data quality monitoring method and device - Google Patents
Data quality monitoring method and device Download PDFInfo
- Publication number
- CN113342791A CN113342791A CN202110599083.0A CN202110599083A CN113342791A CN 113342791 A CN113342791 A CN 113342791A CN 202110599083 A CN202110599083 A CN 202110599083A CN 113342791 A CN113342791 A CN 113342791A
- Authority
- CN
- China
- Prior art keywords
- data quality
- quality monitoring
- field
- monitoring
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 369
- 238000000034 method Methods 0.000 title claims abstract description 61
- 238000012216 screening Methods 0.000 claims abstract description 52
- 238000010801 machine learning Methods 0.000 claims abstract description 26
- 238000012806 monitoring device Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 15
- 230000002159 abnormal effect Effects 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 20
- 238000004891 communication Methods 0.000 description 15
- 230000006870 function Effects 0.000 description 14
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000012545 processing Methods 0.000 description 10
- 238000004364 calculation method Methods 0.000 description 5
- 239000000872 buffer Substances 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 230000001413 cellular effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000000059 patterning Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000017105 transposition Effects 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2457—Query processing with adaptation to user needs
- G06F16/24573—Query processing with adaptation to user needs using data annotations, e.g. user-defined metadata
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Business, Economics & Management (AREA)
- Economics (AREA)
- General Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Library & Information Science (AREA)
- Finance (AREA)
- Marketing (AREA)
- Strategic Management (AREA)
- Technology Law (AREA)
- Computational Linguistics (AREA)
- Computer Hardware Design (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a data quality monitoring method and a data quality monitoring device, belongs to the field of computers, and can be used in the field of finance, and the method comprises the following steps: screening metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list; screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list; determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list; monitoring data quality of monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
Description
Technical Field
The application relates to the field of computers, can be used in the field of finance, and particularly relates to a data quality monitoring method and device.
Background
An important ring in a big data construction system is data quality monitoring. Data quality monitoring systems typically administer their monitored data in batches at intervals. And pushing all data tables which do not do basic monitoring rules (such as primary key uniqueness check and no empty check of the tables) to the responsible person for confirmation, and manually configuring the monitoring rules. Although the above process can be used as a data quality feedback channel to help service personnel to monitor data, the process is cumbersome. And often, the data user will inform the responsible person to perform problem troubleshooting and configure the corresponding rule when finding that the data has problems, and the time is also lagged.
In summary, the drawbacks of the prior art are as follows: firstly, the manual configuration of the data monitoring rule can increase the running time of the whole data link; secondly, the data quality has many problems except the basic monitoring rules, which need to be actively monitored, such as the fluctuation of index data of some fields, and the like, but the prior art does not show any problem.
Disclosure of Invention
Aiming at the problems in the prior art, the application provides a data quality monitoring method and device, which can monitor the data quality by utilizing a machine learning model constructed in advance according to a data quality monitoring rule.
In order to solve the technical problem, the application provides the following technical scheme:
in a first aspect, the present application provides a data quality monitoring method, including:
screening metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list;
determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
monitoring data quality of monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
Further, before determining the data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list, the method further comprises the following steps:
acquiring table information, field dependency relationship and monitoring rule information in the data warehouse;
and constructing the metadata according to the table information, the field dependency relationship and the monitoring rule information.
Further, the screening metadata in the data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list includes:
screening whether each interface table in the data warehouse is empty;
if not, screening whether the table primary key of the interface table is unique;
if the table primary key is unique, screening whether abnormal fields exist in each interface table in the data warehouse or not;
and determining the data quality monitoring table-level list according to the screening result.
Further, the screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list includes:
constructing a field node directed graph according to the table information, the field information and the field dependency relationship;
determining the importance value of each field node in the field node directed graph by using a pre-constructed machine learning model;
and determining a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.
Further, the determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list includes:
respectively acquiring fields in a table-level list according to data quality monitoring and fields in a field-level list according to data quality monitoring;
and (4) taking a union set of the obtained fields to obtain a data quality monitoring output list.
Further, the monitoring information in the data quality monitoring output list is subjected to data quality monitoring, and the monitoring information comprises:
generating a corresponding database description language according to the data quality monitoring rule;
and monitoring the data quality of the monitoring information in the data quality monitoring output list according to the database description language.
In a second aspect, the present application provides a data quality monitoring device, comprising:
the table-level list generating unit is used for screening metadata in the data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
the field-level list generating unit is used for screening the metadata by utilizing a pre-constructed machine learning model to obtain a data quality monitoring field-level list;
the output list generating unit is used for determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
the data quality monitoring unit is used for monitoring the data quality of the monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
Further, the data quality monitoring device further comprises:
the information acquisition unit is used for acquiring table information, field dependency relationship and monitoring rule information in the data warehouse;
and the metadata construction unit is used for constructing the metadata according to the table information, the field dependency relationship and the monitoring rule information.
Further, the table-level list generating unit includes:
the empty table screening module is used for screening whether each interface table in the data warehouse is empty;
the primary key screening module is used for screening whether the table primary key of the interface table is unique if the table primary key is not null;
the abnormal field screening module is used for screening whether abnormal fields exist in each interface table in the data warehouse or not if the table main key is unique;
and the table-level list determining module is used for determining the data quality monitoring table-level list according to the screening result.
Further, the field-level list generating unit includes:
the directed graph construction module is used for constructing a field node directed graph according to the table information, the field information and the field dependency relationship;
the importance value determining module is used for determining the importance value of each field node in the field node directed graph by utilizing a pre-constructed machine learning model;
and the field level list determining module is used for determining a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.
Further, the output list generating unit includes:
the field acquisition module is used for respectively acquiring fields in the data quality monitoring table level list and fields in the data quality monitoring field level list;
and the output list determining module is used for taking a union set of the obtained fields to obtain a data quality monitoring output list.
Further, the data quality monitoring unit includes:
the description language generation module is used for generating a corresponding database description language according to the data quality monitoring rule;
and the data quality monitoring module is used for monitoring the data quality of the monitoring information in the data quality monitoring output list according to the database description language.
In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the data quality monitoring method when executing the program.
In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the data quality monitoring method.
Aiming at the problems in the prior art, the data quality monitoring method and the data quality monitoring device can utilize an input layer, a rule strategy layer, an algorithm layer and an output layer of a data quality monitoring system to realize the full-coverage and more accurate data quality monitoring of a table level and a field level.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of a data quality monitoring method according to an embodiment of the present application;
FIG. 2 is a second flowchart of a data quality monitoring method according to an embodiment of the present application;
FIG. 3 is a flowchart of obtaining a data quality monitoring table-level list in an embodiment of the present application;
FIG. 4 is a flowchart of obtaining a data quality monitoring field level list in an embodiment of the present application;
FIG. 5 is a flow chart of determining a data quality monitoring output list in an embodiment of the present application;
FIG. 6 is a flow chart of data quality monitoring performed in an embodiment of the present application;
FIG. 7 is a block diagram of a data quality monitoring device according to an embodiment of the present disclosure;
FIG. 8 is a second block diagram of a data quality monitoring apparatus according to an embodiment of the present invention;
FIG. 9 is a diagram illustrating a table-level list generating unit according to an embodiment of the present invention;
FIG. 10 is a block diagram of a field level list generation unit in the embodiment of the present application;
fig. 11 is a structural diagram of an output list generation unit in the embodiment of the present application;
FIG. 12 is a block diagram of a data quality monitoring unit in an embodiment of the present application;
fig. 13 is a schematic structural diagram of an electronic device in an embodiment of the present application;
FIG. 14 is a schematic diagram of a data quality monitoring method according to an embodiment of the present application;
FIG. 15 is a second schematic diagram of a data quality monitoring method according to an embodiment of the present application;
FIG. 16 is a third schematic diagram of a data quality monitoring method according to an embodiment of the present application;
fig. 17 is a fourth schematic diagram of a data quality monitoring method in the embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the data quality monitoring method and apparatus provided by the present application may be used in the financial field, and may also be used in any field other than the financial field.
Referring to fig. 1, in order to perform data quality monitoring by using a machine learning model constructed in advance according to a data quality monitoring rule, the present application provides a data quality monitoring method, including:
s101: screening metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
s102: screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list;
it can be understood that the data quality monitoring method provided by the application can be applied to data storage systems in multiple fields such as financial data centers and e-commerce platforms, and the like, and can realize table-level monitoring and field-level monitoring of data quality. The table level monitoring refers to performing general monitoring on each data table stored in the database according to a preset data quality table level monitoring rule. The monitoring granularity of the table level monitoring is relatively coarse, and each data field in the table cannot be monitored deeply. The field-level monitoring is to use a pre-constructed machine learning model to deeply screen each data field in each data table on the basis of table-level monitoring to complete data quality monitoring with relatively fine granularity. The data quality monitoring method is realized based on a graph algorithm, and the realization mode can improve the comprehensiveness and the accuracy of the data quality monitoring rule configuration. The monitoring of the data quality can be realized by relying on a data quality monitoring system, in other words, the data quality monitoring system can be used as an execution main body of the data quality monitoring method provided by the application.
S103: determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
s104: and monitoring the data quality of the monitoring information in the data quality monitoring output list.
It can be understood that, on the basis of the generation of the data quality monitoring table level list and the data quality monitoring field level list, the data quality monitoring system can read the monitoring information in the data quality monitoring table level list and the data quality monitoring field level list, and process and integrate the monitoring information to obtain a data quality monitoring output list; and finally, monitoring the data quality of the monitoring information in the data quality monitoring output list. It should be noted that the monitoring information at least includes a database name, a data table name, a data field name, and an unsatisfied monitoring rule, and in a preferred embodiment, may also include a data quality monitoring person in charge, etc.
The data quality monitoring table level list and the data quality monitoring field level list can be respectively shown as the following tables:
from the above description, the data quality monitoring method provided by the application can utilize the input layer, the rule strategy layer, the algorithm layer and the output layer of the data quality monitoring system to realize the full-coverage and more accurate data quality monitoring at the table level and the field level.
Referring to fig. 2, before determining the data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list, the method further includes:
s201: acquiring table information, field dependency relationship and monitoring rule information in a data warehouse;
s202: and constructing metadata according to the table information, the field dependency relationship and the monitoring rule information.
It is understood that, referring to fig. 17, in the embodiment of the present application, the data quality monitoring system may be divided into four layers, namely, an input layer, a rule policy layer, an algorithm layer, and an output layer. Wherein the process of building metadata may be performed at the input layer.
In particular, the data quality monitoring system may access metadata of a data warehouse or data mart at the input layer. These metadata include, but are not limited to, table information, field dependencies, and monitoring rule information. The metadata can be used to describe basic information, dependency information, call information, and the like of the data, for example, the table information may include a maintainer of the table, a life cycle of the table, a name of the table, a library name to which the table belongs, an access amount of the table, a storage size of the table, and the like; the field information can comprise Chinese description of the field, a table name to which the field belongs, a maintainer of the field, a null value rate of the field, the type of the field and the like; the field dependency relationship can comprise three elements of a field of a parent node (namely, a table name, a library name and a field name), three elements of a child node (namely, a table name, a library name and a field name), a maintainer of the parent node and the child node, a strong and weak dependency type and the like; the monitoring rule information may include a monitoring object, a specific monitoring rule, a monitoring threshold, a monitoring side rule strength, and the like. The metadata can be acquired through automatic collection of a database, manual registration and the like.
For example, one table information may be as follows:
the monitoring rule information may be as follows:
| serial number | Name of storehouse | Table name | Name of field | Rule level | Rule description | ... |
| 1 | A | TABLE1 | / | Meter level | Primary key is unique | ... |
By associating the two metadata according to the database name and the table name, more information can be obtained, such as whether the table is configured with the data monitoring rule and what specific monitoring rule is configured, and the like, so that data support is provided for rule operation of the rule policy layer, and the process can also be understood as a process for constructing the metadata. Of course, the metadata may also be directly obtained from the data warehouse, and the application is not limited thereto.
From the above description, the data quality monitoring method provided by the present application can acquire and construct metadata.
Referring to fig. 3, the metadata in the data warehouse is screened according to the preset data quality monitoring rule to obtain a data quality monitoring table-level list, which includes:
s301: screening whether each interface table in the data warehouse is empty;
s302: when the interface table is not empty, screening whether the table primary key of the interface table is unique;
s303: when the table primary key is unique, screening whether abnormal fields exist in each interface table in the data warehouse or not;
s304: and determining a data quality monitoring table-level list according to the screening result.
S305: and the background records the interface table with the exception.
It can be understood that, the data quality monitoring system in the embodiment of the present application may complete table-level monitoring at the rule policy layer, and pull the data quality monitoring table-level list by configuring the table-level data quality monitoring rule. In the rule policy layer, in the embodiment, a simpler rule can be configured generally, and a data table (interface table) is subjected to preliminary coarser granularity monitoring. These rules allow the selection of data tables (interface tables) that are problematic, i.e., do not meet the table-level data quality monitoring rules, by simply counting the metadata.
For example, the embodiment of the present application recommends configuring the following rules:
interface table must configure the rule that the table is not null
② the interface table must be configured with primary key unique rule
Third, the interface table must configure field level monitoring rules
And fourthly, configuring a null rate fluctuation monitoring rule for the character type field of the interface table (for example, if the null rate of the field 'user's place of residence 'is 10% 7 days ago and the null rate of the field' user's place of residence' is 50% the day, the fluctuation is considered to be too large, and the data quality problem exists).
And fifthly, configuring a total fluctuation monitoring rule for the numerical field of the interface table (for example, if the field is 'the latest transaction amount in 1 year', the sum of all records in the table is 1 hundred million before 1 month, and the sum of all records in the table is 8 hundred million during the current day, the fluctuation is considered to be too large, so that the quality problem exists).
The temporary table and the intermediate table generated in the service are not output to the outside of the service system, and therefore are relatively inferior to the interface table output to the outside of the service system. In order to avoid delaying the throughput time of the data link, in the embodiment of the present application, data quality monitoring is not performed on the data in the temporary table and the intermediate table according to the above rules, and the above data quality monitoring rules are only implemented on the interface table. In a preferred embodiment, the data quality monitoring rules applied to the interface table may also be applied to the temporary table and the intermediate table.
Although fields are involved, the fields need to be monitored on the whole at the level of the table, so that the monitoring needs to be carried out according to the table-level data quality monitoring rule.
As can be seen from the above description, the data quality monitoring method provided by the present application can screen metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list.
Referring to fig. 4, screening metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list, including:
s401: constructing a field node directed graph according to the table information, the field information and the field dependency relationship;
s402: determining the importance value of each field node in the field node directed graph by using a pre-constructed machine learning model;
s403: and determining a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.
It can be understood that the data quality monitoring system in the embodiment of the present application may complete field-level monitoring at a rule policy layer, extract field features by using a graph algorithm, and predict whether a field needs to be monitored by using a machine learning model, where the specific process is as follows:
firstly, patterning: the dependency relationship between fields can be stored in a binary relationship table (table 1), and the binary relationship is converted into a graph relationship by using the idea of a graph. Composition is carried out by taking the library name, the table name and the field name as nodes and taking the dependency relationship as an edge, the graph is a directed graph, and the direction is that a father node with the dependency relationship points to a child node.
For example, a binary relationship as described in Table 1 can be converted to FIG. 14.
TABLE 1 field kindred relationship schematic
| Name of father library | Father table name | Name of father field | Name of child library | Name of sub-table | Name of sub-field |
| A | Table1 | var1 | A | Table2 | Var1 |
| A | Table1 | Var1 | B | Table3 | Var3 |
② calculating importance of nodes
The importance of a node is often computed based on two assumptions: first, the more downstream a node is, the more important this node may be; second, for an important node, its upstream node is often also relatively important. In the embodiment of the application, an improved PageRank algorithm (which can also be understood as reconstructing a machine learning model) is adopted, and a PR value calculated by each node is used as an importance value of the node.
Through the improvement of the applicability of the scene, the embodiment of the application uses the following formula to measure the importance of the node:
piis the current node, pjIs piOne of the nodes downstreamNode, MpiIs piAll downstream sets of nodes, L (p)j) Is pjThe degree of entry of the node.
The PR values of all nodes are iteratively calculated until the result is converged, and then the PR values are normalized to obtain a final value. The convergence threshold value is not specifically limited, and can be specifically selected according to actual conditions.
Fig. 15 illustrates the calculation process of the above algorithm in a simple example, assuming that there is a dependency relationship between nodes as shown in fig. 15.
Then, a matrix may be built from the in-degree case
Divided by the sum of the numbers of the row to yield
By transposition, can obtain
Setting the initial PR value of each node to be 1 to obtain
Then
The iteration continues until X' converges, i.e.:
in the calculation of one iteration, the calculation of the initial calculation,
in the calculation of the iteration once again,
it can be found that X' has converged at this time, and the PR value is normalized to obtain
I.e., the PR values of the three nodes a, b and c are 0.4, 0.2 and 0.4, respectively.
Another example is as follows: dependencies such as fields may be translated into a directed graph as shown in fig. 16.
The following formula can be calculated:
'B':0.027777907029846994
'A':0.2777758071818005
'C':0.06250017033612078
'D':0.07638886092778685
'E':0.06944431583032772
'F':0.04166659762151306
'G':0.027777907029846994
'X':0.027777907029846994
'Y':0.027777907029846994
'Z':0.027777907029846994
'M':0.16666735647660766
'H':0.16666735647660766
the embodiment of the present application may use the above algorithm to calculate, and may also select other algorithms to calculate, which is not limited in this application.
Determining a threshold value: and regarding the nodes exceeding a certain threshold value as needing to configure the monitoring rule, and entering a data quality monitoring field level list. If the threshold is set to 0.07, for the above example, A, M, H, D four nodes are considered as important fields, and the monitoring rule needs to be configured.
As can be seen from the above description, the data quality monitoring method provided by the present application can screen metadata by using a pre-constructed machine learning model to obtain a field-level list of data quality monitoring.
Referring to fig. 5, determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list includes:
s501: respectively acquiring fields in a table-level list according to data quality monitoring and fields in a field-level list according to data quality monitoring;
s502: and (4) taking a union set of the obtained fields to obtain a data quality monitoring output list.
It is understood that the data quality monitoring system may obtain from the data quality monitoring table level list which table needs to be monitored. For the table that needs to be monitored, it can be understood that all the fields therein also need to be monitored. Meanwhile, the data quality monitoring system can also read a data quality monitoring field level list to obtain fields needing quality monitoring. And finally, taking a union set of all the fields to obtain a data quality monitoring output list. The data quality monitoring output list at least comprises but is not limited to monitoring information such as database names, data table names, data field names and unsatisfied monitoring rules.
As can be seen from the above description, the data quality monitoring method provided in the present application can determine the data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list.
Referring to fig. 6, the data quality monitoring of the monitoring information in the data quality monitoring output list includes:
s601: generating a corresponding database description language according to the data quality monitoring rule;
s602: and monitoring the data quality of the monitoring information in the data quality monitoring output list according to the database description language.
It is to be appreciated that the data quality monitoring system can generate a corresponding database description language according to the data quality monitoring rules. For example, when configuring the table-level data quality monitoring rules, the screening unique to the primary key may correspond to the database description language as follows. Among them, Structured Query Language (SQL) is more commonly used.
Select t1. library name, t1. table name, t1. field name, t1. maintainer, 'primary key unique' as missing rule description
From
(Select from table metadata where table type ═ interface table') t1
Left outer join
(Select from monitoring rule metadata where field name = '/' and rule description ═
'Primary Key unique') t2
On t1. library name t2. library name and t1. table name t2. table name
where t2. table name is null
The data table with non-unique main keys can be screened out through the database description language, and other data quality monitoring rules can be analogized. And finally, monitoring the data quality of the monitoring information in the data quality monitoring output list according to the database description language.
As can be seen from the above description, the data quality monitoring method provided in the present application can perform data quality monitoring on the monitoring information in the data quality monitoring output list.
Based on the same inventive concept, the embodiment of the present application further provides a data quality monitoring apparatus, which can be used to implement the methods described in the above embodiments, as described in the following embodiments. Because the principle of the data quality monitoring device for solving the problems is similar to that of the data quality monitoring method, the implementation of the data quality monitoring device can refer to the implementation of the software performance reference determination method, and repeated parts are not described again. As used hereinafter, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. While the system described in the embodiments below is preferably implemented in software, implementations in hardware, or a combination of software and hardware are also possible and contemplated.
Referring to fig. 7, in order to perform data quality monitoring using a machine learning model constructed in advance according to a data quality monitoring rule, the present application provides a data quality monitoring apparatus, including:
a table-level list generating unit 701, configured to screen metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
a field-level list generating unit 702, configured to screen the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field-level list;
an output list generating unit 703, configured to determine a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
a data quality monitoring unit 704, configured to perform data quality monitoring on monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
Referring to fig. 8, the data quality monitoring apparatus further includes:
an information obtaining unit 801, configured to obtain table information, field dependency, and monitoring rule information in the data warehouse;
a metadata constructing unit 802, configured to construct the metadata according to the table information, the field dependency relationship, and the monitoring rule information.
Referring to fig. 9, the table-level manifest generating unit 701 includes:
an empty table screening module 901, configured to screen whether each interface table in the data warehouse is empty;
a primary key screening module 902, configured to screen whether a table primary key of the interface table is unique if the table primary key is not null;
an abnormal field screening module 903, configured to screen whether an abnormal field exists in each interface table in the data warehouse if the table primary key is unique;
and a table-level list determining module 904, configured to determine the data quality monitoring table-level list according to the screening result.
Referring to fig. 10, the field level list generation unit 702 includes:
a directed graph construction module 1001 configured to construct a field node directed graph according to the table information, the field information, and the field dependency relationship;
an importance value determining module 1002, configured to determine an importance value of each field node in the field node directed graph by using a pre-constructed machine learning model;
a field level list determining module 1003, configured to determine a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.
Referring to fig. 11, the output list generation unit 703 includes:
a field obtaining module 1101, configured to obtain fields in a table-level list according to data quality monitoring and fields in a field-level list according to data quality monitoring;
and an output list determining module 1102, configured to obtain a data quality monitoring output list by merging the obtained fields.
Referring to fig. 12, the data quality monitoring unit 704 includes:
a description language generation module 1201, configured to generate a corresponding database description language according to the data quality monitoring rule;
and the data quality monitoring module 1202 is configured to perform data quality monitoring on the monitoring information in the data quality monitoring output list according to the database description language.
In terms of hardware, in order to perform data quality monitoring by using a pre-constructed machine learning model according to a data quality monitoring rule, the present application provides an embodiment of an electronic device for implementing all or part of the contents in the data quality monitoring method, where the electronic device specifically includes the following contents:
a Processor (Processor), a Memory (Memory), a communication Interface (Communications Interface) and a bus; the processor, the memory and the communication interface complete mutual communication through the bus; the communication interface is used for realizing information transmission between the data quality monitoring device and relevant equipment such as a core service system, a user terminal, a relevant database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, and the like, but the embodiment is not limited thereto. In this embodiment, the logic controller may be implemented with reference to the embodiments of the data quality monitoring method and the data quality monitoring apparatus in the embodiments, and the contents thereof are incorporated herein, and repeated descriptions thereof are omitted.
It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), an in-vehicle device, a smart wearable device, and the like. Wherein, intelligence wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..
In practical applications, part of the data quality monitoring method may be performed on the electronic device side as described above, or all operations may be performed in the client device. The selection may be specifically performed according to the processing capability of the client device, the limitation of the user usage scenario, and the like. This is not a limitation of the present application. The client device may further include a processor if all operations are performed in the client device.
The client device may have a communication module (i.e., a communication unit), and may be in communication connection with a remote server to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementation scenarios, the server may also include a server on an intermediate platform, for example, a server on a third-party server platform that is communicatively linked to the task scheduling center server. The server may include a single computer device, or may include a server cluster formed by a plurality of servers, or a server structure of a distributed apparatus.
Fig. 13 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 13, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 13 is exemplary; other types of structures may also be used in addition to or in place of the structure to implement telecommunications or other functions.
In one embodiment, the data quality monitoring method functions may be integrated into the central processor 9100. The central processor 9100 may be configured to control as follows:
s101: screening metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
s102: screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list;
s103: determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
s104: monitoring data quality of monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
From the above description, the data quality monitoring method provided by the application can utilize the input layer, the rule strategy layer, the algorithm layer and the output layer of the data quality monitoring system to realize the full-coverage and more accurate data quality monitoring at the table level and the field level.
In another embodiment, the data quality monitoring device may be configured separately from the central processing unit 9100, for example, the data quality monitoring device of the data composite transmission apparatus may be configured as a chip connected to the central processing unit 9100, and the function of the data quality monitoring method may be implemented by the control of the central processing unit.
As shown in fig. 13, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 also does not necessarily include all of the components shown in fig. 13; in addition, the electronic device 9600 may further include components not shown in fig. 13, which can be referred to in the prior art.
As shown in fig. 13, a central processor 9100, sometimes referred to as a controller or operational control, can include a microprocessor or other processor device and/or logic device, which central processor 9100 receives input and controls the operation of the various components of the electronic device 9600.
The memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information relating to the failure may be stored, and a program for executing the information may be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to realize information storage or processing, or the like.
The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. Power supply 9170 is used to provide power to electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, an LCD display, but is not limited thereto.
The memory 9140 can be a solid state memory, e.g., Read Only Memory (ROM), Random Access Memory (RAM), a SIM card, or the like. There may also be a memory that holds information even when power is off, can be selectively erased, and is provided with more data, an example of which is sometimes called an EPROM or the like. The memory 9140 could also be some other type of device. Memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 being used for storing application programs and function programs or for executing a flow of operations of the electronic device 9600 by the central processor 9100.
The memory 9140 can also include a data store 9143, the data store 9143 being used to store data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, contact book applications, etc.).
The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, which may be the same as in the case of a conventional mobile communication terminal.
Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless lan module, may be disposed in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby implementing ordinary telecommunications functions. The audio processor 9130 may include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling recording locally through the microphone 9132 and enabling locally stored sounds to be played through the speaker 9131.
An embodiment of the present application further provides a computer-readable storage medium capable of implementing all the steps in the data quality monitoring method with the execution subject being the server or the client in the foregoing embodiments, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps in the data quality monitoring method with the execution subject being the server or the client in the foregoing embodiments, for example, when the processor executes the computer program, the processor implements the following steps:
s101: screening metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
s102: screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list;
s103: determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
s104: monitoring data quality of monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
From the above description, the data quality monitoring method provided by the application can utilize the input layer, the rule strategy layer, the algorithm layer and the output layer of the data quality monitoring system to realize the full-coverage and more accurate data quality monitoring at the table level and the field level.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The principle and the implementation mode of the invention are explained by applying specific embodiments in the invention, and the description of the embodiments is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.
Claims (14)
1. A method for monitoring data quality, comprising:
screening metadata in a data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list;
determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
monitoring data quality of monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
2. The data quality monitoring method according to claim 1, before determining the data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list, further comprising:
acquiring table information, field dependency relationship and monitoring rule information in the data warehouse;
and constructing the metadata according to the table information, the field dependency relationship and the monitoring rule information.
3. The data quality monitoring method according to claim 2, wherein the screening metadata in the data warehouse according to the preset data quality monitoring rule to obtain the data quality monitoring table-level list comprises:
screening whether each interface table in the data warehouse is empty;
if not, screening whether the table primary key of the interface table is unique;
if the table primary key is unique, screening whether abnormal fields exist in each interface table in the data warehouse or not;
and determining the data quality monitoring table-level list according to the screening result.
4. The data quality monitoring method according to claim 2, wherein the screening the metadata by using a pre-constructed machine learning model to obtain a data quality monitoring field level list comprises:
constructing a field node directed graph according to the table information, the field information and the field dependency relationship;
determining the importance value of each field node in the field node directed graph by using a pre-constructed machine learning model;
and determining a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.
5. The method according to claim 1, wherein determining the data quality monitoring output list according to the data quality monitoring table-level list and the data quality monitoring field-level list comprises:
respectively acquiring fields in a table-level list according to data quality monitoring and fields in a field-level list according to data quality monitoring;
and (4) taking a union set of the obtained fields to obtain a data quality monitoring output list.
6. The data quality monitoring method according to claim 1, wherein the performing data quality monitoring on the monitoring information in the data quality monitoring output list comprises:
generating a corresponding database description language according to the data quality monitoring rule;
and monitoring the data quality of the monitoring information in the data quality monitoring output list according to the database description language.
7. A data quality monitoring device, comprising:
the table-level list generating unit is used for screening metadata in the data warehouse according to a preset data quality monitoring rule to obtain a data quality monitoring table-level list;
the field-level list generating unit is used for screening the metadata by utilizing a pre-constructed machine learning model to obtain a data quality monitoring field-level list;
the output list generating unit is used for determining a data quality monitoring output list according to the data quality monitoring table level list and the data quality monitoring field level list;
the data quality monitoring unit is used for monitoring the data quality of the monitoring information in the data quality monitoring output list; the monitoring information at least comprises a database name, a data table name, a data field name and an unsatisfied monitoring rule.
8. The data quality monitoring device of claim 7, further comprising:
the information acquisition unit is used for acquiring table information, field dependency relationship and monitoring rule information in the data warehouse;
and the metadata construction unit is used for constructing the metadata according to the table information, the field dependency relationship and the monitoring rule information.
9. The data quality monitoring device according to claim 8, wherein the table-level list generating unit includes:
the empty table screening module is used for screening whether each interface table in the data warehouse is empty;
the primary key screening module is used for screening whether the table primary key of the interface table is unique if the table primary key is not null;
the abnormal field screening module is used for screening whether abnormal fields exist in each interface table in the data warehouse or not if the table main key is unique;
and the table-level list determining module is used for determining the data quality monitoring table-level list according to the screening result.
10. The data quality monitoring device according to claim 8, wherein the field-level list generation unit includes:
the directed graph construction module is used for constructing a field node directed graph according to the table information, the field information and the field dependency relationship;
the importance value determining module is used for determining the importance value of each field node in the field node directed graph by utilizing a pre-constructed machine learning model;
and the field level list determining module is used for determining a data quality monitoring field level list according to a preset importance threshold and the importance value of each field node.
11. The data quality monitoring device according to claim 7, wherein the output list generation unit includes:
the field acquisition module is used for respectively acquiring fields in the data quality monitoring table level list and fields in the data quality monitoring field level list;
and the output list determining module is used for taking a union set of the obtained fields to obtain a data quality monitoring output list.
12. The data quality monitoring device according to claim 7, wherein the data quality monitoring unit comprises:
the description language generation module is used for generating a corresponding database description language according to the data quality monitoring rule;
and the data quality monitoring module is used for monitoring the data quality of the monitoring information in the data quality monitoring output list according to the database description language.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the data quality monitoring method according to any one of claims 1 to 6 are implemented when the program is executed by the processor.
14. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the data quality monitoring method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110599083.0A CN113342791A (en) | 2021-05-31 | 2021-05-31 | Data quality monitoring method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110599083.0A CN113342791A (en) | 2021-05-31 | 2021-05-31 | Data quality monitoring method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113342791A true CN113342791A (en) | 2021-09-03 |
Family
ID=77472441
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110599083.0A Pending CN113342791A (en) | 2021-05-31 | 2021-05-31 | Data quality monitoring method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113342791A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116108021A (en) * | 2023-04-11 | 2023-05-12 | 北方健康医疗大数据科技有限公司 | Method, device and equipment for performing multi-dimensional data quality control based on standard data set |
| CN116402480A (en) * | 2023-06-07 | 2023-07-07 | 成都普朗克科技有限公司 | Method and system for outputting inventory based on association rule self-built model |
| WO2024039017A1 (en) * | 2022-08-16 | 2024-02-22 | Samsung Electronics Co., Ltd. | Method and apparatus for managing quality of data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708909A (en) * | 2015-11-18 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Data quality detection method and apparatus |
| CN107515886A (en) * | 2016-06-17 | 2017-12-26 | 阿里巴巴集团控股有限公司 | A kind of recognition methods of tables of data, device and system |
| CN110098961A (en) * | 2019-04-25 | 2019-08-06 | 北京天融信网络安全技术有限公司 | A kind of Data Quality Assessment Methodology, device and storage medium |
| WO2020207014A1 (en) * | 2019-04-09 | 2020-10-15 | 平安科技（深圳）有限公司 | Automated regression testing method and apparatus for big data, test center server and storage medium |
| CN111913954A (en) * | 2020-06-20 | 2020-11-10 | 杭州城市大数据运营有限公司 | Intelligent data standard catalog generation method and device |
| CN112084269A (en) * | 2018-12-25 | 2020-12-15 | 北京锐安科技有限公司 | Data quality calculation method and device, storage medium and server |
-
2021
- 2021-05-31 CN CN202110599083.0A patent/CN113342791A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708909A (en) * | 2015-11-18 | 2017-05-24 | 阿里巴巴集团控股有限公司 | Data quality detection method and apparatus |
| CN107515886A (en) * | 2016-06-17 | 2017-12-26 | 阿里巴巴集团控股有限公司 | A kind of recognition methods of tables of data, device and system |
| CN112084269A (en) * | 2018-12-25 | 2020-12-15 | 北京锐安科技有限公司 | Data quality calculation method and device, storage medium and server |
| WO2020207014A1 (en) * | 2019-04-09 | 2020-10-15 | 平安科技（深圳）有限公司 | Automated regression testing method and apparatus for big data, test center server and storage medium |
| CN110098961A (en) * | 2019-04-25 | 2019-08-06 | 北京天融信网络安全技术有限公司 | A kind of Data Quality Assessment Methodology, device and storage medium |
| CN111913954A (en) * | 2020-06-20 | 2020-11-10 | 杭州城市大数据运营有限公司 | Intelligent data standard catalog generation method and device |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2024039017A1 (en) * | 2022-08-16 | 2024-02-22 | Samsung Electronics Co., Ltd. | Method and apparatus for managing quality of data |
| CN116108021A (en) * | 2023-04-11 | 2023-05-12 | 北方健康医疗大数据科技有限公司 | Method, device and equipment for performing multi-dimensional data quality control based on standard data set |
| CN116108021B (en) * | 2023-04-11 | 2023-09-08 | 北方健康医疗大数据科技有限公司 | Method, device and equipment for performing multi-dimensional data quality control based on standard data set |
| CN116402480A (en) * | 2023-06-07 | 2023-07-07 | 成都普朗克科技有限公司 | Method and system for outputting inventory based on association rule self-built model |
| CN116402480B (en) * | 2023-06-07 | 2023-09-19 | 成都普朗克科技有限公司 | Method and system for outputting inventory based on association rule self-built model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113342791A (en) | Data quality monitoring method and device | |
| Bertsimas et al. | Adaptive distributionally robust optimization | |
| CN109377329B (en) | House resource recommendation method and device, storage medium and electronic equipment | |
| US11249969B2 (en) | Data storage method and apparatus, and storage medium | |
| CN106802932B (en) | Routing method and device of database and database system | |
| US10909484B2 (en) | Dynamic directed graph workflows | |
| US20220020064A1 (en) | Feature processing method and apparatus for artificial intelligence recommendation model, electronic device, and storage medium | |
| US8244760B2 (en) | Segmentation and profiling of users | |
| US9977819B2 (en) | Sharing data on mobile devices | |
| US20130339293A1 (en) | Methods and systems for flexible and scalable databases | |
| WO2014116527A1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| KR20160054017A (en) | Bottom-up optimized search system and method | |
| CN113590576A (en) | Database parameter adjusting method and device, storage medium and electronic equipment | |
| CN103678591A (en) | Device and method for automatically executing multi-service receipt statistical treatment | |
| CN106844319B (en) | Report generation method and device | |
| US20210334312A1 (en) | System and method for generating highly scalable temporal graph database | |
| CN107861933A (en) | The method and apparatus for generating O＆M form | |
| CN108804454B (en) | Group portrait method, group portrait device and server | |
| CN111488531B (en) | Information recommendation method, device and medium based on collaborative filtering algorithm | |
| CN110309147B (en) | Bidirectional tracing method and device for aircraft on-orbit test | |
| CN110019456B (en) | Data import method, device and system | |
| CN112734404A (en) | Iterative man-hour statistical method and device in research and development process and computer equipment | |
| CN109670980B (en) | Method, system and equipment for processing charging service data | |
| CN105653205A (en) | User request processing method and system | |
| CN111951011A (en) | Monitoring system threshold determination method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |