CN116644078A - Data quality inspection method, inspection device, inspection equipment and storage medium - Google Patents

Data quality inspection method, inspection device, inspection equipment and storage medium Download PDF

Info

Publication number
CN116644078A
CN116644078A CN202310619769.0A CN202310619769A CN116644078A CN 116644078 A CN116644078 A CN 116644078A CN 202310619769 A CN202310619769 A CN 202310619769A CN 116644078 A CN116644078 A CN 116644078A
Authority
CN
China
Prior art keywords
data
checked
data quality
application system
standard
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310619769.0A
Other languages
Chinese (zh)
Inventor
郭群
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Bank of China Ltd
Original Assignee
Bank of China Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Bank of China Ltd filed Critical Bank of China Ltd
Priority to CN202310619769.0A priority Critical patent/CN116644078A/en
Publication of CN116644078A publication Critical patent/CN116644078A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G06F16/2365—Ensuring data consistency and integrity
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computer Security & Cryptography (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • General Factory Administration (AREA)

Abstract

The application discloses a data quality inspection method, an inspection device, inspection equipment and a storage medium, which can be applied to the financial field or other fields, and the method comprises the following steps: determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system; determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in a data dictionary according to a one-to-one correspondence; inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and checking the data quality of the data to be checked; acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; and the same data standard is used for checking the data quality of the same data, so that the consistency of the data quality in the service system database is ensured.

Description

Data quality inspection method, inspection device, inspection equipment and storage medium
Technical Field
The present application relates to the field of data processing, and in particular, to a data quality inspection method, an inspection apparatus, an inspection device, and a storage medium.
Background
The data quality is the guarantee of the business data effectively managed by enterprises, and the data quality inspection of the business data is an effective means for guaranteeing the data quality.
In the related art, the data quality inspection is to make data quality inspection rules according to the understanding of the data standards by system developers and write the data quality inspection rules into a data inspection program, and the problem that the quality inspection results of the data have deviation due to inconsistent quality inspection rules exists at present.
Disclosure of Invention
The embodiment of the application provides a data quality inspection method, an inspection device, inspection equipment and a storage medium, which improve the consistency of data quality in an application system.
In view of this, a first aspect of the present application provides a data quality inspection method, the method comprising:
determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system;
determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in the data dictionary according to a one-to-one correspondence;
inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and performing data quality check on the data to be checked;
acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality check result is used for describing whether the data to be checked meets the data standard.
Optionally, the data quality inspection model is trained by:
acquiring training data from the application system database, and acquiring the data standard corresponding to the training data from the data dictionary;
when the training data meets the data standard, marking the training data as compliance data to obtain a training data set;
and training a machine learning classification model based on the training set to obtain the data quality inspection model.
Optionally, the method further comprises:
and updating the data quality check model based on the updated data dictionary when the data dictionary is updated.
Optionally, the method further comprises:
setting a data quality check period;
inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and performing data quality check on the data to be checked comprises:
and inputting the data to be checked and the data standard into a data quality check model corresponding to the application system every other data quality check period, and performing data quality check on the data to be checked.
A second aspect of the present application provides a data quality inspection apparatus, the apparatus comprising:
a data determination unit configured to: determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system;
a standard determining unit for: determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in the data dictionary according to a one-to-one correspondence;
a data quality checking unit for: inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and performing data quality check on the data to be checked;
a result acquisition unit configured to: acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality check result is used for describing whether the data to be checked meets the data standard.
Optionally, the apparatus further comprises a model training unit for:
acquiring training data from the application system database, and acquiring the data standard corresponding to the training data from the data dictionary;
when the training data meets the data standard, marking the training data as compliance data to obtain a training data set;
and training a machine learning classification model based on the training set to obtain the data quality inspection model.
Optionally, the apparatus further comprises a model updating unit for:
and updating the data quality check model based on the updated data dictionary when the data dictionary is updated.
Optionally, the apparatus further comprises a period determining unit for:
setting a data quality check period;
the data quality inspection unit is specifically configured to:
and inputting the data to be checked and the data standard into a data quality check model corresponding to the application system every other data quality check period, and performing data quality check on the data to be checked.
A third aspect of the present application provides a data quality inspection apparatus, the apparatus comprising: a memory and a processor;
the memory is used for storing instructions;
the processor is configured to execute the instructions in the memory and perform the method described above.
A fourth aspect of the application provides a computer readable storage medium storing program code or instructions which, when run on a computer, cause the computer to perform the method described above.
From the above technical scheme, the application has the following advantages: determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system; determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in a data dictionary according to a one-to-one correspondence; the data dictionary stores the data standard corresponding to each data, and the consistency of the data standards is improved based on the unified management of the data standards; inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and checking the data quality of the data to be checked; acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality inspection result is used for describing whether the data to be inspected accords with the data standard, and the same data standard is used for data quality inspection on the same data, so that the consistency of the data quality inspection rule is improved, and the consistency of the data quality in the service system database is further ensured.
Drawings
FIG. 1 is a flowchart of a data quality inspection method according to an embodiment of the present application;
FIG. 2 is a flowchart of a training method for a data quality inspection model according to an embodiment of the present application;
FIG. 3 is a system architecture diagram for performing a data quality inspection method according to an embodiment of the present application;
FIG. 4 is a block diagram of a data quality inspection device according to an embodiment of the present application;
fig. 5 is a block diagram of a data quality inspection apparatus according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in more detail below with reference to the accompanying drawings. While the application is susceptible of embodiment in the drawings, it is to be understood that the application may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided to provide a more thorough and complete understanding of the application. It should be understood that the drawings and embodiments of the application are for illustration purposes only and are not intended to limit the scope of the present application.
Referring to fig. 1, an embodiment of the present application provides a data quality inspection method, which specifically includes the following steps:
step 101: the data to be checked is determined from the application database.
When data quality inspection is performed on data stored in the application system database, data to be inspected in the data quality inspection can be determined from the application system database according to actual requirements of the data quality inspection, the data to be inspected can be one piece of data in the application system database, can be multiple pieces of data in the application system database, and can be all data in the application system database.
Step 102: and determining the data standard corresponding to the data to be checked from the data dictionary according to the data name of the data to be checked.
The data names and the data standards are stored in the data dictionary according to a one-to-one correspondence. In a data dictionary, a data standard is a specification formulated for achieving the goals of data consistency, comparability, maintainability and the like, and relates to various specifications of naming rules, data structures, data types, data formats, data acquisition methods, data storage modes and the like, and when data quality inspection is carried out, the formulation and use of the data standard is an effective means for guaranteeing data consistency, standardizing management and improving data quality. In the embodiment of the application, each data standard comprises at least parameters including a data name, a data structure, a data type and the like, the data standards are stored in a data dictionary according to the data name, and one data name corresponds to one data standard. And searching a data standard corresponding to the data name from the data dictionary according to the data name of the data to be checked, wherein the data standard is the data standard corresponding to the data to be checked.
Step 103: and inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and checking the data quality of the data to be checked.
After the data to be checked of the application system and the data standard corresponding to the data to be checked are obtained, the data is input into a data quality check model corresponding to the application system, the data quality check is carried out on the data to be checked through the data quality check model, and whether the data to be checked accords with the corresponding data standard is judged. In different application system databases, the stored data content may not be the same, so that a data quality check model corresponding to different application systems needs to be built according to the data quality check model aiming at the application systems.
The data quality inspection model is obtained by training a machine learning classification model based on a training set, as shown in fig. 2, and in one possible implementation, the data quality inspection model is trained by the following method:
step 201: acquiring training data from an application system database, and acquiring data standards corresponding to the training data from a data dictionary;
step 202: when the training data meets the data standard, marking the training data as compliance data to obtain a training data set;
step 203: and training the machine learning classification model based on the training set to obtain a data quality inspection model.
The method comprises the steps of collecting a large amount of training data from an application system database, determining data standards corresponding to the training data one by one in a data dictionary according to the data names of the training data, marking the training data according to the data standards, marking the training data as compliant data when the training data accords with the data standards corresponding to the training data, otherwise, marking the training data as non-compliant data, generating a training data set according to the marked training data, and training a machine learning classification model based on the training data set to obtain a data quality inspection model. The data quality inspection model is obtained based on a large amount of data training in the training data set, so that accuracy of a data quality inspection result obtained in the data quality inspection process is improved, and data quality inspection efficiency is improved.
Step 104: and acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model.
In the data quality inspection model, outputting a data quality inspection result based on data to be inspected and a data standard, wherein the data quality inspection result is used for describing whether the data to be inspected accords with the data standard, and outputting the data quality inspection result that the data to be inspected is the compliance data when the data to be inspected accords with the data standard; when the data to be checked does not accord with the data standard, outputting a data quality check result that the data to be checked is non-compliance data, wherein the data quality check result of the non-compliance data can also comprise reasons for the non-compliance of the data to be checked, such as reasons for data type errors, data structure errors and the like. Furthermore, the data to be checked can be corrected according to the reason that the data to be checked is not compliant, and the data to be checked is converted into data conforming to the data standard.
The embodiment of the application determines the data to be checked from the application system database; the application system database stores data generated in the running process of the application system; determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in a data dictionary according to a one-to-one correspondence; the data dictionary stores the data standard corresponding to each data, and the consistency of the data standards is improved based on the unified management of the data standards; inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and checking the data quality of the data to be checked; acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality inspection result is used for describing whether the data to be inspected accords with the data standard, and the same data standard is used for data quality inspection on the same data, so that the consistency of the data quality inspection rule is improved, and the consistency of the data quality in the service system database is further ensured.
In one possible implementation, the method shown in fig. 1 may further include the following steps:
when the data dictionary is updated, the data quality inspection model is updated based on the updated data dictionary.
Along with the continuous development of enterprises, the data dictionary used in the data quality inspection process is updated continuously, and the data quality inspection model corresponding to each application system is updated according to the updated data dictionary because the data quality inspection model needs to use the data standard in the data dictionary when the data quality inspection is performed. To improve the performance of the data quality inspection model and the accuracy of the data quality inspection result.
In one possible implementation, the method shown in fig. 1 may further include the following steps:
a data quality check period is set.
When the data quality inspection is carried out on the data in the application system database, different data quality inspection periods can be set for the data quality inspection according to the data quality requirement, and when the data quality requirement on the application system database is higher, a shorter data quality inspection period is set for ensuring the data quality in the application system database; when the data quality requirement on the application system database is lower, a longer data quality check period is set, so that the data resources occupied when the data quality check flow is frequently triggered are reduced.
Step 103 may be implemented at this time by:
and inputting the data to be inspected and the data standard into a data quality inspection model corresponding to the application system every other data quality inspection period, and performing data quality inspection on the data to be inspected.
By setting the data quality inspection period, the data quality inspection is automatically carried out on the data in the application system database, so that the time and resources for manually triggering the data quality inspection flow are saved, and the efficiency of the data quality inspection is improved.
Fig. 3 is a system architecture diagram for performing the data quality inspection method shown in fig. 1 according to an embodiment of the present application. In fig. 3, the data dictionary management module is configured to manage a data dictionary, and synchronize the data dictionary to the model automatic construction module, where the data dictionary includes parameters such as a data dictionary name, a service type, a data type, a security level, and the like; the model automatic construction module constructs a data quality inspection model corresponding to the application system according to the data in the databases of different application systems and the corresponding data standards of the data in the data dictionary, for example, the data quality inspection model is respectively constructed for the application system 1 and the application system 2 in fig. 3, and the model is used for inspecting whether the data value range deviation, the data storage mode, the data length, the data precision and the like meet the requirements of the data standards or not and outputting the corresponding data quality inspection result. When the data dictionary is updated, the model automatic construction module automatically updates the data quality inspection model through the synchronized data dictionary and pushes the new version of the data quality inspection model to the corresponding application system; after the data quality inspection is finished, the data quality inspection result can be output to the application system for the operation and maintenance manager to check.
Although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous.
It should be understood that the various steps recited in the method embodiments of the present application may be performed in a different order and/or performed in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the application is not limited in this respect.
Referring to fig. 4, an embodiment of the present application provides a data quality inspection apparatus, including: a data determination unit 401, a standard determination unit 402, a data quality inspection unit 403, and a result acquisition unit 404.
The data determination unit 401 is configured to: determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system;
the standard determining unit 402 is configured to: determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in a data dictionary according to a one-to-one correspondence;
the data quality checking unit 403 is configured to: inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and checking the data quality of the data to be checked;
the result acquisition unit 404 is configured to: acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality check result is used to describe whether the data to be checked meets the data standard.
In a possible implementation, the apparatus shown in fig. 4 further includes a model training unit configured to:
acquiring training data from an application system database, and acquiring data standards corresponding to the training data from a data dictionary;
when the training data meets the data standard, marking the training data as compliance data to obtain a training data set;
and training the machine learning classification model based on the training set to obtain a data quality inspection model.
In a possible implementation manner, the apparatus shown in fig. 4 further includes a model updating unit, configured to:
when the data dictionary is updated, the data quality inspection model is updated based on the updated data dictionary.
In a possible implementation, the apparatus shown in fig. 4 further includes a period determining unit configured to:
setting a data quality check period;
the data quality checking unit 403 is specifically configured to:
and inputting the data to be inspected and the data standard into a data quality inspection model corresponding to the application system every other data quality inspection period, and performing data quality inspection on the data to be inspected.
Referring to fig. 5, an embodiment of the present application provides a data quality inspection apparatus, including: a memory 501 and a processor 502;
a memory 501 for storing instructions;
a processor 502 for executing instructions in the memory 501, performing the above method.
Embodiments of the present application provide a computer readable storage medium storing program code or instructions that, when run on a computer, cause the computer to perform the above method.
It should be noted that the data quality inspection method, the inspection device, the inspection apparatus and the storage medium provided by the present application may be used in the financial field or other fields, for example, may be used in a data quality inspection scenario in the financial field. Other fields are any field other than the financial field, for example, the data processing field. The foregoing is merely exemplary, and does not limit the application fields of the data quality inspection method, the inspection apparatus, the inspection device and the storage medium provided by the present application.
The names of messages or information interacted between the devices in the embodiments of the present application are for illustrative purposes only and are not intended to limit the scope of such messages or information.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a usb disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media in which a computer program can be stored.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims (10)

1. A method of data quality inspection, the method comprising:
determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system;
determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in the data dictionary according to a one-to-one correspondence;
inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and performing data quality check on the data to be checked;
acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality check result is used for describing whether the data to be checked meets the data standard.
2. The method of claim 1, wherein the data quality inspection model is trained by:
acquiring training data from the application system database, and acquiring the data standard corresponding to the training data from the data dictionary;
when the training data meets the data standard, marking the training data as compliance data to obtain a training data set;
and training a machine learning classification model based on the training set to obtain the data quality inspection model.
3. The method according to claim 1, wherein the method further comprises:
and updating the data quality check model based on the updated data dictionary when the data dictionary is updated.
4. The method according to claim 1, wherein the method further comprises:
setting a data quality check period;
inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and performing data quality check on the data to be checked comprises:
and inputting the data to be checked and the data standard into a data quality check model corresponding to the application system every other data quality check period, and performing data quality check on the data to be checked.
5. A data quality inspection device, the device comprising:
a data determination unit configured to: determining data to be checked from an application system database; the application system database stores data generated in the running process of the application system;
a standard determining unit for: determining a data standard corresponding to the data to be checked from a data dictionary according to the data name of the data to be checked; the data names and the data standards are stored in the data dictionary according to a one-to-one correspondence;
a data quality checking unit for: inputting the data to be checked and the data standard into a data quality check model corresponding to the application system, and performing data quality check on the data to be checked;
a result acquisition unit configured to: acquiring a data quality inspection result corresponding to the data to be inspected from the data quality inspection model; the data quality check result is used for describing whether the data to be checked meets the data standard.
6. The apparatus of claim 5, further comprising a model training unit for:
acquiring training data from the application system database, and acquiring the data standard corresponding to the training data from the data dictionary;
when the training data meets the data standard, marking the training data as compliance data to obtain a training data set;
and training a machine learning classification model based on the training set to obtain the data quality inspection model.
7. The apparatus according to claim 5, further comprising a model updating unit configured to:
and updating the data quality check model based on the updated data dictionary when the data dictionary is updated.
8. The apparatus according to claim 5, further comprising a period determination unit configured to:
setting a data quality check period;
the data quality inspection unit is specifically configured to:
and inputting the data to be checked and the data standard into a data quality check model corresponding to the application system every other data quality check period, and performing data quality check on the data to be checked.
9. A data quality inspection apparatus, the apparatus comprising: a memory and a processor;
the memory is used for storing instructions;
the processor being configured to execute the instructions in the memory and to perform the method of any of claims 1-4.
10. A computer readable storage medium storing program code or instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1-4 above.
CN202310619769.0A 2023-05-29 2023-05-29 Data quality inspection method, inspection device, inspection equipment and storage medium Pending CN116644078A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310619769.0A CN116644078A (en) 2023-05-29 2023-05-29 Data quality inspection method, inspection device, inspection equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310619769.0A CN116644078A (en) 2023-05-29 2023-05-29 Data quality inspection method, inspection device, inspection equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116644078A true CN116644078A (en) 2023-08-25

Family

ID=87614923

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310619769.0A Pending CN116644078A (en) 2023-05-29 2023-05-29 Data quality inspection method, inspection device, inspection equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116644078A (en)

Similar Documents

Publication Publication Date Title
CN110309071B (en) Test code generation method and module, and test method and system
US8799869B2 (en) System for ensuring comprehensiveness of requirements testing of software applications
CN111722839B (en) Code generation method and device, electronic equipment and storage medium
CN109359277B (en) Data monitoring method, device and computer storage medium
CN107908550B (en) Software defect statistical processing method and device
CN111767340B (en) Data processing method, device, electronic equipment and medium
CN113360519B (en) Data processing method, device, equipment and storage medium
CN107766353B (en) Method and device for migrating statistical information of database
CN112416778A (en) Test case recommendation method and device and electronic equipment
CN111259040A (en) SQL statement auditing method and system
CN111143390A (en) Method and device for updating metadata
CN112559525B (en) Data checking system, method, device and server
CN112069052A (en) Abnormal object detection method, device, equipment and storage medium
CN113760891A (en) Data table generation method, device, equipment and storage medium
CN107273293B (en) Big data system performance test method and device and electronic equipment
CN116644078A (en) Data quality inspection method, inspection device, inspection equipment and storage medium
CN113791980B (en) Conversion analysis method, device and equipment for test cases and storage medium
CN113435830A (en) Mail information summarizing method, system, electronic device and storage medium
CN111859985A (en) AI customer service model testing method, device, electronic equipment and storage medium
CN111767222A (en) Data model verification method and device, electronic equipment and storage medium
CN116661758B (en) Method, device, electronic equipment and medium for optimizing log framework configuration
CN112035425B (en) Log storage method and device and computer system
CN113495831B (en) Method, system, equipment and medium for generating test case based on keywords
CN115470112A (en) Extraction method of reusable test case, server and storage medium
CN117707943A (en) Data regression testing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination