CN112148721B - Data checking method and device, electronic equipment and storage medium - Google Patents

Data checking method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112148721B
CN112148721B CN202011021590.8A CN202011021590A CN112148721B CN 112148721 B CN112148721 B CN 112148721B CN 202011021590 A CN202011021590 A CN 202011021590A CN 112148721 B CN112148721 B CN 112148721B
Authority
CN
China
Prior art keywords
data
check
rule
table corresponding
checking
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011021590.8A
Other languages
Chinese (zh)
Other versions
CN112148721A (en
Inventor
元立本
石超
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
New H3C Big Data Technologies Co Ltd
Original Assignee
New H3C Big Data Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by New H3C Big Data Technologies Co Ltd filed Critical New H3C Big Data Technologies Co Ltd
Priority to CN202011021590.8A priority Critical patent/CN112148721B/en
Publication of CN112148721A publication Critical patent/CN112148721A/en
Application granted granted Critical
Publication of CN112148721B publication Critical patent/CN112148721B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution
    • G06F16/24564—Applying rules; Deductive queries

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a data checking method, a data checking device, electronic equipment and a storage medium, wherein a preset monitor is used for recording data change conditions of a checking table and a reference table corresponding to each checking rule; in the process of data checking of the first checking rule which is not executed for the first time, acquiring the data change conditions of a checking table and a reference table corresponding to the first checking rule and the last checking result of the first checking rule; and obtaining the current checking result of the first checking rule according to the data change conditions of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule. The current checking result of the first checking rule is obtained by using the data change condition of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule, all data in the checking table and the reference table are not required to be obtained from the data source during each checking, the reading and writing pressure of the data source server can be reduced, and the data checking efficiency is improved.

Description

Data checking method and device, electronic equipment and storage medium
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data checking method and apparatus, an electronic device, and a storage medium.
Background
Today, with the development of information-based applications, data is one of the most valuable assets. The data quality of the enterprise and the business performance are directly connected, and the high-quality data can improve the competitiveness of the enterprise. How to process mass data and acquire high-quality data has become a focus of attention of various industries. Data quality analysis supports monitoring of service indexes and data quality, data can be analyzed from six dimensions of completeness, effectiveness, timeliness, accuracy and uniqueness, data quality inspection can help users to find data quality problems in time, correct problems in time and reduce loss. However, today with large information explosion, the types of data and the amount of data become larger and larger, and data quality analysis faces the problem of inefficient performance of checking out valid data from massive data.
In the prior art, data is acquired from a check table of a data source in a JDBC (Java Database Connectivity) manner, and then, on the condition of the data in the check table, whether the same data exists is searched in a traversal reference table in a JDBC manner, and if the same data does not exist, the data in the check table is returned to a monitoring end as error data, thereby implementing data check.
However, by using the above data checking method, each time the checking task is executed, all data in the checking table and the reference table need to be repeatedly acquired from the data source, the read-write pressure of the data source server is large, and the data checking efficiency is low.
Disclosure of Invention
An embodiment of the present application aims to provide a data checking method and apparatus, an electronic device, and a storage medium, so as to reduce read-write pressure of a data source server and increase data checking efficiency. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a data checking method, where the method includes:
recording the data change conditions of the check tables and the reference tables corresponding to each check rule by using a preset monitor, wherein one check rule corresponds to at least one check table and at least one reference table;
in the process of data checking of a first checking rule which is not executed for the first time, acquiring data change conditions of a checking table and a reference table corresponding to the first checking rule and a last checking result of the first checking rule, wherein the first detecting rule is any checking rule;
and obtaining a current checking result of the first checking rule according to the data change conditions of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule.
In a possible implementation manner, before the recording, by using a preset listener, data change conditions of the check table and the reference table corresponding to each check rule, the method further includes:
in the process of performing the data check of the first check rule for the first time, acquiring and comparing the total data in the check table corresponding to the first check rule with the total data in the reference table to obtain a first check result of the first check rule, wherein the first check result of the first check rule comprises error data in the check table corresponding to the first check rule;
storing the error data in the corresponding checking table of the first checking rule into a specified storage;
the recording of the data change conditions of the check table and the reference table corresponding to each check rule by using the preset monitor comprises the following steps:
and recording the data change conditions of the check table and the reference table corresponding to the first check rule by using a preset monitor.
In a possible implementation manner, the obtaining, according to the data change condition of the check table and the reference table corresponding to the first check rule and the last check result of the first check rule, the current check result of the first check rule includes:
updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table;
and obtaining a current checking result of the first checking rule according to the error data of the checking table corresponding to the first checking rule recorded in the designated storage.
In one possible embodiment, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table includes:
comparing the target error data in the specified storage with the data added in the reference table corresponding to the first checking rule to obtain the same data as first data to be deleted when the data change condition indicates that the data in the reference table corresponding to the first checking rule is increased, wherein the target error data in the specified storage is the error data in the checking table corresponding to the first checking rule recorded in the specified storage;
and deleting the first data to be deleted from the error data of the check table corresponding to the first check rule of the appointed storage record.
In one possible embodiment, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table includes:
when the data change situation indicates that the data in the reference table corresponding to the first check rule is reduced, comparing the reduced data in the reference table corresponding to the first check rule with the full data in the check table corresponding to the first check rule to obtain the same data as the first data to be updated;
and adding the first data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
In one possible embodiment, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table includes:
comparing the data before modification in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule under the condition that the data change condition indicates that the data in the reference table corresponding to the first checking rule are modified, and obtaining the same data as second data to be updated;
comparing the modified data in the reference table corresponding to the first checking rule with the target error data in the designated storage to obtain the same data serving as second data to be deleted, wherein the target error data in the designated storage is the error data in the checking table corresponding to the first checking rule recorded in the designated storage;
and adding the second data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the second data to be deleted.
In one possible embodiment, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table includes:
comparing the target error data in the specified storage with the reduced data in the check table corresponding to the first check rule to obtain the same data as third data to be deleted when the data change condition indicates that the data in the check table corresponding to the first check rule is reduced, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and deleting the third data to be deleted in the error data of the checking table corresponding to the first checking rule of the specified storage record.
In one possible embodiment, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table includes:
comparing the data added in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is added, and obtaining wrong data serving as third data to be updated;
and adding the third data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
In one possible embodiment, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table includes:
comparing the modified data in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is modified, and obtaining wrong data serving as fourth data to be updated;
comparing the data before modification in the check table corresponding to the first check rule with the target error data in the designated storage to obtain the same data as fourth data to be deleted, wherein the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage;
and adding the fourth data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the fourth data to be deleted.
In a second aspect, an embodiment of the present application provides a data checking apparatus, where the apparatus includes:
the data change recording module is used for recording the data change conditions of the check tables and the reference tables corresponding to all check rules by using a preset monitor, wherein one check rule corresponds to at least one check table and at least one reference table;
the data change acquisition module is used for acquiring the data change conditions of a check table and a reference table corresponding to a first check rule and the last check result of the first check rule in the process of performing the data check of the first check rule for a non-first time;
and the checking result determining module is used for obtaining the current checking result of the first checking rule according to the data change conditions of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule.
In one possible embodiment, the apparatus further comprises:
the system comprises a first checking rule, a full data comparison module and a second checking rule, wherein the first checking rule is used for acquiring and comparing full data in a checking table corresponding to the first checking rule and full data in a reference table in the process of performing data checking of the first checking rule for the first time to obtain a first checking result of the first checking rule, and the first checking result of the first checking rule comprises error data in the checking table corresponding to the first checking rule;
the error data storage module is used for storing the error data in the corresponding check table of the first check rule into a specified storage;
the data change recording module is specifically configured to: and recording the data change conditions of the check table and the reference table corresponding to the first check rule by using a preset monitor.
In a possible implementation manner, the check result determining module includes:
the error data updating submodule is used for updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change conditions of the check table corresponding to the first check rule and the reference table;
and the checking result obtaining submodule is used for obtaining the current checking result of the first checking rule according to the error data of the checking table corresponding to the first checking rule recorded in the designated storage.
In a possible implementation, the error data update submodule is specifically configured to:
comparing the target error data in the specified storage with the data added in the reference table corresponding to the first checking rule to obtain the same data as first data to be deleted when the data change condition indicates that the data in the reference table corresponding to the first checking rule is increased, wherein the target error data in the specified storage is the error data in the checking table corresponding to the first checking rule recorded in the specified storage;
and deleting the first data to be deleted from the error data of the check table corresponding to the first check rule of the appointed storage record.
In a possible implementation, the error data update sub-module is specifically configured to:
comparing the reduced data in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule to obtain the same data as the first data to be updated when the data change condition indicates that the data in the reference table corresponding to the first checking rule is reduced;
and adding the first data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation, the error data update submodule is specifically configured to:
comparing the data before modification in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule under the condition that the data change condition indicates that the data in the reference table corresponding to the first checking rule are modified, and obtaining the same data as second data to be updated;
comparing the modified data in the reference table corresponding to the first check rule with the target error data in the specified storage to obtain the same data as second data to be deleted, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and adding the second data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the second data to be deleted.
In a possible implementation, the error data update submodule is specifically configured to:
comparing the target error data in the specified storage with the reduced data in the check table corresponding to the first check rule to obtain the same data as third data to be deleted when the data change condition indicates that the data in the check table corresponding to the first check rule is reduced, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and deleting the third data to be deleted in the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation, the error data update sub-module is specifically configured to:
comparing the data added in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is added, and obtaining wrong data serving as third data to be updated;
and adding the third data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation, the error data update sub-module is specifically configured to:
comparing the modified data in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is modified, and obtaining wrong data serving as fourth data to be updated;
comparing the data before modification in the check table corresponding to the first check rule with the target error data in the designated storage to obtain the same data as fourth data to be deleted, wherein the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage;
and adding the fourth data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the fourth data to be deleted.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement any of the data checking methods when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements any one of the data checking methods.
The data checking method, the data checking device, the electronic equipment and the storage medium provided by the embodiment of the application record the data change conditions of the checking table and the reference table corresponding to each checking rule by using the preset monitor, wherein one checking rule corresponds to at least one checking table and at least one reference table; in the process of data checking of a first checking rule which is not executed for the first time, acquiring the data change conditions of a checking table and a reference table corresponding to the first checking rule and the last checking result of the first checking rule; and obtaining the current checking result of the first checking rule according to the data change conditions of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule. The current checking result of the first checking rule is obtained by using the data change condition of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule, all data in the checking table and the reference table are not required to be obtained from the data source every time of checking, the read-write pressure of the data source server can be reduced, the comparison between the total data in the checking table and the total data in the reference table is not required to be carried out every time of checking, and the data checking efficiency can be improved. Of course, it is not necessary for any product or method of the present application to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a first schematic diagram of a data checking method according to an embodiment of the present application;
FIG. 2 is a second schematic diagram of a data checking method according to an embodiment of the present application;
FIG. 3 is a schematic diagram illustrating an implementation manner of step S13 according to an embodiment of the present application;
FIG. 4 is a first schematic diagram illustrating an implementation manner of step S131 in the embodiment of the present application;
fig. 5 is a second schematic diagram of an implementation manner of step S131 in the embodiment of the present application;
FIG. 6 is a third schematic diagram illustrating an implementation manner of step S131 in the embodiment of the present application;
FIG. 7 is a fourth schematic diagram illustrating an implementation manner of step S131 in the embodiment of the present application;
FIG. 8 is a fifth schematic diagram illustrating an implementation manner of step S131 in the embodiment of the present application;
fig. 9 is a sixth schematic diagram of an implementation manner of step S131 in the embodiment of the present application;
FIG. 10 is a schematic diagram of a data checking device according to an embodiment of the present application;
fig. 11 is a schematic view of an electronic device according to an embodiment of the application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
First, terms in the present application are introduced:
checking a table: a table for analyzing mass data is required.
Referring to the table: and a table is established in practical application as standard data.
Redis: an open-source, memory-based, and persistent journaling, key-Value database.
Spark: a fast general purpose computing engine designed specifically for large scale data processing.
In order to reduce the read-write pressure of the data source server and increase the data checking efficiency, an embodiment of the present application provides a data checking method, and referring to fig. 1, the method includes:
and S11, recording the data change condition of the check table and the reference table corresponding to each check rule by using a preset monitor, wherein one check rule corresponds to at least one check table and at least one reference table.
The data verification method of the embodiment of the application can be realized through a server, the server can be a single server or a server cluster formed by a plurality of servers, and the average value is within the protection range of the application.
The checking rules can be created according to actual conditions in a self-defined mode, the checking rules correspond to the checking tasks, one checking task comprises at least one checking rule, and the checking tasks can take the checking rules as basic execution units. A checking rule corresponds to at least one checking table and at least one reference table. The preset monitor can be a monitor thread based on Redis, monitors the data change conditions of the check table and the reference table corresponding to each check rule by using the preset monitor, and records the data change conditions of the check table and the reference table corresponding to each check rule.
S12, during the data check process without first executing the first check rule, obtains the data change condition of the check table and the reference table corresponding to the first check rule, and the last check result of the first check rule.
The first check rule is any check rule; for each checking rule, the same operation as the first checking rule may be performed. In the process of executing the first check rule for a non-first time (i.e. not for the first time), a preset monitor is called to obtain the data change condition of the check table and the reference table corresponding to the first check rule, wherein the data change condition records the data change condition of the check table and the reference table corresponding to the first check rule after the last check, including data addition, deletion and modification.
And S13, obtaining the current checking result of the first checking rule according to the data change condition of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule.
And determining the wrong data in the check table corresponding to the first check rule after last check according to the data change conditions of the check table corresponding to the first check rule and the reference table, and obtaining the check result of the first check rule at this time by combining the check result of the first check rule last time (last time).
In the embodiment of the application, the check result of the first check rule is obtained by using the data change condition of the check table and the reference table corresponding to the first check rule and the last check result of the first check rule, all data in the check table and the reference table are not required to be obtained from the data source every time of checking, the read-write pressure of the data source server can be reduced, the comparison between the total data in the check table and the total data in the reference table is not required to be carried out every time of checking, and the data check efficiency can be improved.
In a possible implementation manner, referring to fig. 2, before the recording, by using a preset listener, data changes of the check table and the reference table corresponding to each check rule, the method further includes:
and S14, in the process of performing the data check of the first check rule for the first time, obtaining and comparing the total data in the check table corresponding to the first check rule with the total data in the reference table to obtain a first check result of the first check rule, where the first check result of the first check rule includes the error data in the check table corresponding to the first check rule.
The total data in the check table corresponding to the first check rule and the total data in the reference table corresponding to the first check rule can be inquired in batch through a Spark calculation engine which is created in advance, and the total data in the check table corresponding to the first check rule and the total data in the reference table corresponding to the first check rule are compared, so that the error data in the check table corresponding to the first check rule is obtained. Optionally, the error data in the corresponding check table of the first check rule may be: data that is present in the corresponding check table of the first check rule but not in the corresponding look-up table of the first check rule.
S15, storing the error data in the corresponding check table of the first check rule in a designated storage.
Designated as a preset data block or folder, etc. For example, in the case where the pre-set listener is a Redis-based listener thread, the designated store may be a Redis database.
The above-mentioned data change condition that utilizes the corresponding check table of each rule of checking of presetting listener record and reference table includes:
and S111, recording the data change conditions of the check table and the reference table corresponding to the first check rule by using a preset monitor.
After the first check rule is executed for the first time, a preset monitor can be used for monitoring and recording the data change conditions of the check table and the reference table corresponding to the first check rule. For example, a to-be-monitored check rule table may be maintained, the check rule after being executed for the first time is added to the to-be-monitored check rule table, and the preset monitor monitors and records data change conditions of the check table and the reference table corresponding to each check rule in the to-be-monitored check rule table.
In the embodiment of the application, when the data check of the first check rule is executed for the first time, the check result of the first check rule is obtained by adopting a full data comparison mode, and the preset monitor is used for recording the data change conditions of the check table and the reference table corresponding to the first check rule after the first data check, so that the check result of the first check rule at the current time is obtained by using the data change conditions and the last check result, all data in the check table and the reference table are not required to be obtained from the data source during each check, the read-write pressure of the data source server can be reduced, and the data check efficiency is increased.
In one possible embodiment, referring to fig. 3, the obtaining of the current checking result of the first checking rule according to the data change of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule includes:
s131, updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change of the check table and the reference table corresponding to the first check rule.
And according to the data change conditions of the check table corresponding to the first check rule and the reference table, obtaining the error data which is increased or decreased in the check table corresponding to the first check rule, so as to update the error data of the check table corresponding to the first check rule recorded in the appointed storage.
And S132, obtaining the current checking result of the first checking rule according to the error data of the checking table corresponding to the first checking rule recorded in the appointed storage.
After the updating of the error data in the check table corresponding to the first check rule recorded in the designated storage is completed, the error data of the check table corresponding to the first check rule recorded in the storage is obtained, namely the error data in the current check result of the first check rule.
In the embodiment of the present application, by updating the error data of the check table recorded in the designated storage, the data check efficiency can be increased.
The data change comprises three conditions of data addition, modification and reduction (deletion). Next, a description will be given of an update of error data in the check table corresponding to the first check rule recorded in the designated storage.
In one possible embodiment, referring to fig. 4, the updating the error data of the check table corresponding to the first checking rule recorded in the designated storage according to the data change condition of the check table corresponding to the first checking rule and the reference table includes:
s131a1, when the data change indicates that data in the reference table corresponding to the first check rule is increased, comparing the target error data in the designated storage with the data increased in the reference table corresponding to the first check rule to obtain the same data as the first data to be deleted, where the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage.
The target error data is the error data in the checking table corresponding to the first core rule recorded in a designated storage, such as a Redis database. If it is specified that the stored target error data does not have the same data as the data added to the reference table corresponding to the first check rule, step S131a2 does not have to be executed.
S131a2, deleting the first data to be deleted from the error data in the check table corresponding to the first check rule of the designated storage record.
If the first data to be deleted does not exist in the error data of the check table corresponding to the first check rule specifying the storage record, step S131a2 does not have to be executed.
In the embodiment of the application, a mode of updating the error data recorded in the designated storage by using the data added in the reference table is disclosed, and the updating of the error data recorded in the designated storage can be realized when the data in the reference table is increased.
In one possible embodiment, referring to fig. 5, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change of the check table corresponding to the first check rule and the reference table includes:
s131b1, when the data change indicates that the data in the reference table corresponding to the first check rule is decreased, comparing the decreased data in the reference table corresponding to the first check rule with the total data in the check table corresponding to the first check rule, and obtaining the same data as the first data to be updated.
Optionally, the reduced data in the reference table corresponding to the first check rule is compared with the full data in the check table corresponding to the first check rule, where, in the full data in the check table corresponding to the first check rule, the data that is the same as the reduced data in the reference table corresponding to the first check rule is the wrong data. For example, if the reduced data in the reference table corresponding to the first checking rule is AAA, and the total data in the checking table corresponding to the first checking rule is BBB, ABA, AAA, BAA, or BAB, the AAA in the checking table corresponding to the first checking rule is used as the first data to be updated.
If the reduced data in the reference table corresponding to the first check rule does not have the same data as the full data in the check table corresponding to the first check rule, step S131b2 does not have to be executed.
S131b2, adding the first to-be-updated data to the error data of the check table corresponding to the first check rule of the specified storage record.
In the embodiment of the application, a mode of updating error data recorded in a designated storage by using reduced data in a reference table is disclosed, and the updating of the error data recorded in the designated storage can be realized when the data in the reference table is reduced.
In one possible embodiment, referring to fig. 6, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change of the check table corresponding to the first check rule and the reference table includes:
s131c1, if the data change indicates that the data in the reference table corresponding to the first check rule is modified, comparing the data before modification in the reference table corresponding to the first check rule with the full data in the check table corresponding to the first check rule, and obtaining the same data as the second data to be updated.
Optionally, the data before modification in the reference table corresponding to the first check rule is compared with the full data in the check table corresponding to the first check rule, where in the full data in the check table corresponding to the first check rule, the data that is the same as the data before modification in the reference table corresponding to the first check rule is the second data to be updated. For example, if the modified data in the reference table corresponding to the first checking rule is that AAA is modified to ABA, AAA is data before modification; and if the total data in the checking table corresponding to the first checking rule are BBB, ABA, AAA, BAA and BAB, taking the AAA in the checking table corresponding to the first checking rule as the second data to be updated.
S131c2, comparing the modified data in the reference table corresponding to the first check rule with the target error data in the specified storage, and obtaining the same data as the second data to be deleted, where the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage.
Optionally, the modified data in the reference table corresponding to the first check rule is compared with the target error data in the designated storage, where, in the target error data in the designated storage, the data that is the same as the modified data in the reference table corresponding to the first check rule is the second data to be deleted. For example, the modified data in the reference table corresponding to the first checking rule is that AAA is modified to ABA, and then ABA is modified data; and if the target error data recorded in the appointed storage is BBB, ABA, AAA, BAA and BAB, the ABA in the target error data recorded in the appointed storage is used as the second data to be deleted.
S131c3, adding the second data to be updated to the error data in the check table corresponding to the first check rule of the designated storage record, and deleting the second data to be deleted.
In the embodiment of the application, a mode of updating the error data recorded in the designated storage by using the modified data in the reference table is disclosed, and the updating of the error data recorded in the designated storage can be realized when the data in the reference table is modified.
In one possible embodiment, referring to fig. 7, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change of the check table corresponding to the first check rule and the reference table includes:
s131d1, when the data change indicates that the data in the check table corresponding to the first check rule is decreased, comparing the target error data in the designated storage with the decreased data in the check table corresponding to the first check rule to obtain the same data as the third data to be deleted, where the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage.
If it is specified that the target error data in the storage does not have the same data as the reduced data in the check table corresponding to the first check rule, step S131d2 does not have to be executed.
S131d2, deleting the third data to be deleted from the error data in the check table corresponding to the first check rule of the designated storage record.
If the third data to be deleted does not exist in the error data of the check table corresponding to the first check rule specifying the storage record, step S131d2 does not have to be executed.
In the embodiment of the application, a mode of updating the error data recorded in the designated storage by using the reduced data in the check table is disclosed, and the updating of the error data recorded in the designated storage can be realized when the data in the check table is reduced.
In one possible embodiment, referring to fig. 8, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change of the check table corresponding to the first check rule and the reference table includes:
s131e1, when the data change indicates that the data in the check table corresponding to the first check rule is increased, comparing the data increased in the check table corresponding to the first check rule with the total data in the reference table corresponding to the first check rule, and obtaining error data as third data to be updated.
Optionally, the data added to the check table corresponding to the first check rule is compared with the full data in the reference table corresponding to the first check rule, wherein if the same data as the data added to the check table corresponding to the first check rule does not exist in the full data in the reference table corresponding to the first check rule, the data added to the check table corresponding to the first check rule is error data. For example, the data added to the checking table corresponding to the first checking rule is AAC, the full data in the reference table corresponding to the first checking rule is BBB, ABA, AAA, BAA, and BAB, and if there is no AAC, the data added to the checking table corresponding to the first checking rule is AAC error data, that is, the third data to be updated.
If the added data in the check table corresponding to the first check rule does not have the same data as the total data in the reference table corresponding to the first check rule, step S131e2 does not need to be executed.
S131e2, adding the third data to be updated to the error data in the check table corresponding to the first check rule of the designated storage record.
In the embodiment of the application, a mode of updating the error data recorded in the designated storage by using the data added in the check table is disclosed, and the updating of the error data recorded in the designated storage can be realized when the data in the check table is increased.
In one possible embodiment, referring to fig. 9, the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change of the check table corresponding to the first check rule and the reference table includes:
s131f1, when the data change condition indicates that the data in the check table corresponding to the first check rule is modified, comparing the modified data in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule, and obtaining the wrong data as the fourth data to be updated.
Optionally, the modified data in the check table corresponding to the first check rule is compared with the full data in the reference table corresponding to the first check rule, wherein if the same data as the modified data in the check table corresponding to the first check rule does not exist in the full data in the reference table corresponding to the first check rule, the modified data in the check table corresponding to the first check rule is error data, that is, fourth data to be updated. For example, if the modified data in the checking table corresponding to the first checking rule is that AAA is modified into AAC, AAC is modified data; the total data in the reference table corresponding to the first check rule are BBB, ABA, AAA, BAA and BAB, and if AAC does not exist, the modified data AAC in the check table corresponding to the first check rule is error data.
S131f2, comparing the data before modification in the check table corresponding to the first check rule with the target error data in the designated storage, to obtain the same data as the fourth data to be deleted, where the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage.
Optionally, the data before modification in the check table corresponding to the first check rule is compared with the target error data in the designated storage, where in the target error data in the designated storage, the data that is the same as the data before modification in the check table corresponding to the first check rule is the fourth data to be deleted. For example, if the modified data in the checking table corresponding to the first checking rule is that AAA is modified to AAC, AAA is data before modification; and if the target error data recorded in the storage are designated as BBB, ABA, AAA, BAA and BAB, the AAA in the target error data recorded in the designated storage is taken as fourth data to be deleted.
S131f3, adding the fourth data to be updated to the error data in the check table corresponding to the first check rule of the designated storage record, and deleting the fourth data to be deleted.
In the embodiment of the application, a mode of updating the error data recorded in the specified storage by using the modified data in the check table is disclosed, and the error data recorded in the specified storage can be updated when the data in the check table is modified.
It is understood that when the data change situation indicates that there are multiple changes and/or there are changes in the check table reference table at the same time, the embodiments shown in fig. 4 to 9 may be combined according to the actual change situation, so as to update the error data in the check table corresponding to the first check rule recorded in the designated storage.
For example, in the case where the data in the check table corresponding to the first check rule is decreased and the data in the reference table corresponding to the first check rule is modified, the error data in the check table corresponding to the first check rule recorded in the designated storage may be updated in accordance with the embodiments shown in fig. 7 and 6.
For example, in the case where part of the data is reduced in the check table corresponding to the first check rule and another part of the data is added in the check table corresponding to the first check rule, the embodiment shown in fig. 7 is used for the reduced data to update the error data in the check table corresponding to the first check rule recorded in the designated storage; the embodiment shown in fig. 8 is adopted for the added data to update the error data in the check table corresponding to the first check rule recorded in the designated storage.
An embodiment of the present application further provides a data checking apparatus, referring to fig. 10, the apparatus includes:
the data change recording module 101 is configured to record, by using a preset listener, data change conditions of check tables and reference tables corresponding to each check rule, where one check rule corresponds to at least one check table and at least one reference table;
a data change obtaining module 102, configured to obtain, in a process of performing data checking of a first checking rule for a non-first time, data change conditions of a checking table and a reference table corresponding to the first checking rule, and a last checking result of the first checking rule;
and the checking result determining module 103 is configured to obtain a current checking result of the first checking rule according to the data change condition of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule.
In a possible embodiment, the above apparatus further comprises:
a total data comparison module, configured to, in a process of performing data check of the first check rule for the first time, obtain and compare total data in a check table corresponding to the first check rule with total data in a reference table, to obtain a first check result of the first check rule, where the first check result of the first check rule includes error data in the check table corresponding to the first check rule;
an error data storage module, configured to store the error data in the check table corresponding to the first check rule in a designated storage;
the data change recording module is specifically configured to: and recording the data change conditions of the check table and the reference table corresponding to the first check rule by using a preset monitor.
In a possible implementation manner, the check result determining module includes:
an error data updating submodule, configured to update the error data in the check table corresponding to the first check rule recorded in the designated storage according to the data change conditions of the check table corresponding to the first check rule and the reference table;
and the checking result obtaining submodule is used for obtaining the current checking result of the first checking rule according to the error data of the checking table corresponding to the first checking rule recorded in the appointed storage.
In a possible implementation manner, the error data update sub-module is specifically configured to:
comparing the target error data in the designated storage with the data added in the reference table corresponding to the first checking rule to obtain the same data as first data to be deleted when the data change condition indicates that the data in the reference table corresponding to the first checking rule is increased, wherein the target error data in the designated storage is the error data in the checking table corresponding to the first checking rule recorded in the designated storage;
and deleting the first data to be deleted from the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation manner, the error data update sub-module is specifically configured to:
when the data change situation indicates that the data in the reference table corresponding to the first check rule is reduced, comparing the reduced data in the reference table corresponding to the first check rule with the total data in the check table corresponding to the first check rule to obtain the same data as the first data to be updated;
and adding the first data to be updated to the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation manner, the error data update sub-module is specifically configured to:
comparing the data before modification in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule under the condition that the data change condition indicates that the data in the reference table corresponding to the first checking rule are modified, and obtaining the same data as second data to be updated;
comparing the modified data in the reference table corresponding to the first check rule with the target error data in the specified storage to obtain the same data as second data to be deleted, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and adding the second data to be updated to the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the second data to be deleted.
In a possible implementation manner, the error data update sub-module is specifically configured to:
comparing the target error data in the designated storage with the reduced data in the check table corresponding to the first check rule to obtain the same data as third data to be deleted when the data change condition indicates that the data in the check table corresponding to the first check rule is reduced, wherein the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage;
and deleting the third data to be deleted from the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation manner, the error data update sub-module is specifically configured to:
comparing the data added in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule when the data change condition indicates that the data in the check table corresponding to the first check rule is added, and obtaining error data serving as third data to be updated;
and adding the third data to be updated to the error data of the check table corresponding to the first check rule of the specified storage record.
In a possible implementation manner, the error data update sub-module is specifically configured to:
comparing the modified data in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is modified, and obtaining wrong data serving as fourth data to be updated;
comparing the data before modification in the check table corresponding to the first check rule with the target error data in the specified storage to obtain the same data as fourth data to be deleted, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and adding the fourth data to be updated to the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the fourth data to be deleted.
An embodiment of the present application further provides an electronic device, including: a processor and a memory;
the memory is used for storing computer programs;
the processor is configured to implement any of the above data checking methods when executing the computer program stored in the memory.
Optionally, referring to fig. 11, the electronic device according to the embodiment of the present application further includes a communication interface 202 and a communication bus 204, where the processor 201, the communication interface 202, and the memory 203 complete communication with each other through the communication bus 204.
The communication bus mentioned in the electronic device may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this is not intended to represent only one bus or type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a RAM (Random Access Memory) or an NVM (Non-Volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements any of the data checking methods described above:
in yet another embodiment provided by the present application, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the above-described data checking methods.
In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optic, digital subscriber line) or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It should be noted that, in this document, the technical features in the various alternatives can be combined to form the scheme as long as the technical features are not contradictory, and the scheme is within the scope of the disclosure of the present application. Relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on differences from other embodiments. In particular, for the embodiments of the apparatus, the electronic device, and the storage medium, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the scope of protection of the present application.

Claims (16)

1. A method of data inspection, the method comprising:
in the process of performing data check of a first check rule for the first time, acquiring and comparing total data in a check table corresponding to the first check rule and total data in a reference table to obtain a first check result of the first check rule, wherein the first check result of the first check rule comprises error data in the check table corresponding to the first check rule;
storing the error data in the corresponding checking table of the first checking rule into a specified storage;
the method for recording the data change conditions of the check table and the reference table corresponding to each check rule by using the preset monitor comprises the following steps: recording the data change conditions of the check table and the reference table corresponding to the first check rule by using a preset monitor, wherein one check rule corresponds to at least one check table and at least one reference table;
in the process of data checking of a first checking rule which is not executed for the first time, acquiring data change conditions of a checking table and a reference table corresponding to the first checking rule and a last checking result of the first checking rule;
obtaining a current checking result of the first checking rule according to the data change conditions of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule, including: updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change conditions of the check table corresponding to the first check rule and the reference table; and obtaining a current checking result of the first checking rule according to the error data of the checking table corresponding to the first checking rule recorded in the designated storage.
2. The method according to claim 1, wherein the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table comprises:
comparing the target error data in the specified storage with the data added in the reference table corresponding to the first checking rule under the condition that the data change condition indicates that the data in the reference table corresponding to the first checking rule is increased, and obtaining the same data as first data to be deleted, wherein the target error data in the specified storage is the error data in the checking table corresponding to the first checking rule recorded in the specified storage;
and deleting the first data to be deleted from the error data of the check table corresponding to the first check rule of the appointed storage record.
3. The method according to claim 1, wherein the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table comprises:
comparing the reduced data in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule to obtain the same data as the first data to be updated when the data change condition indicates that the data in the reference table corresponding to the first checking rule is reduced;
and adding the first data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
4. The method according to claim 1, wherein the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table comprises:
comparing the data before modification in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule under the condition that the data change condition indicates that the data in the reference table corresponding to the first checking rule are modified, and obtaining the same data as second data to be updated;
comparing the modified data in the reference table corresponding to the first checking rule with the target error data in the designated storage to obtain the same data serving as second data to be deleted, wherein the target error data in the designated storage is the error data in the checking table corresponding to the first checking rule recorded in the designated storage;
and adding the second data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the second data to be deleted.
5. The method according to claim 1, wherein the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table comprises:
comparing the target error data in the specified storage with the reduced data in the check table corresponding to the first check rule to obtain the same data as third data to be deleted when the data change condition indicates that the data in the check table corresponding to the first check rule is reduced, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and deleting the third data to be deleted in the error data of the checking table corresponding to the first checking rule of the specified storage record.
6. The method according to claim 1, wherein the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table comprises:
comparing the data added in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is added, and obtaining wrong data serving as third data to be updated;
and adding the third data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
7. The method according to claim 1, wherein the updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change condition of the check table corresponding to the first check rule and the reference table comprises:
comparing the modified data in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is modified, and obtaining wrong data serving as fourth data to be updated;
comparing the data before modification in the check table corresponding to the first check rule with the target error data in the designated storage to obtain the same data as fourth data to be deleted, wherein the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage;
and adding the fourth data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the fourth data to be deleted.
8. A data checking apparatus, characterized in that the apparatus comprises:
the system comprises a full data comparison module, a first check rule processing module and a second check rule processing module, wherein the full data comparison module is used for acquiring and comparing full data in a check table corresponding to a first check rule and full data in a reference table in the process of executing data check of the first check rule for the first time to obtain a first check result of the first check rule, and the first check result of the first check rule comprises error data in the check table corresponding to the first check rule;
the error data storage module is used for storing the error data in the corresponding check table of the first check rule into a specified storage;
the data change recording module is used for recording the data change conditions of the check table and the reference table corresponding to each check rule by using a preset monitor, and comprises the following steps: recording the data change conditions of the check table and the reference table corresponding to the first check rule by using a preset monitor, wherein one check rule corresponds to at least one check table and at least one reference table;
the data change acquisition module is used for acquiring the data change conditions of a check table and a reference table corresponding to a first check rule and the last check result of the first check rule in the process of performing the data check of the first check rule for a non-first time;
the checking result determining module is used for obtaining the current checking result of the first checking rule according to the data change condition of the checking table and the reference table corresponding to the first checking rule and the last checking result of the first checking rule;
the checking result determining module comprises:
the error data updating submodule is used for updating the error data of the check table corresponding to the first check rule recorded in the designated storage according to the data change conditions of the check table corresponding to the first check rule and the reference table;
and the checking result obtaining submodule is used for obtaining the current checking result of the first checking rule according to the error data of the checking table corresponding to the first checking rule recorded in the specified storage.
9. The apparatus of claim 8, wherein the error data update sub-module is specifically configured to:
comparing the target error data in the specified storage with the data added in the reference table corresponding to the first checking rule to obtain the same data as first data to be deleted when the data change condition indicates that the data in the reference table corresponding to the first checking rule is increased, wherein the target error data in the specified storage is the error data in the checking table corresponding to the first checking rule recorded in the specified storage;
and deleting the first data to be deleted in the error data of the check table corresponding to the first check rule of the specified storage record.
10. The apparatus of claim 8, wherein the error data update sub-module is specifically configured to:
comparing the reduced data in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule to obtain the same data as the first data to be updated when the data change condition indicates that the data in the reference table corresponding to the first checking rule is reduced;
and adding the first data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
11. The apparatus of claim 8, wherein the error data update submodule is specifically configured to:
comparing the data before modification in the reference table corresponding to the first checking rule with the full data in the checking table corresponding to the first checking rule under the condition that the data change condition indicates that the data in the reference table corresponding to the first checking rule are modified, and obtaining the same data as second data to be updated;
comparing the modified data in the reference table corresponding to the first checking rule with the target error data in the designated storage to obtain the same data serving as second data to be deleted, wherein the target error data in the designated storage is the error data in the checking table corresponding to the first checking rule recorded in the designated storage;
and adding the second data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the second data to be deleted.
12. The apparatus of claim 8, wherein the error data update submodule is specifically configured to:
comparing the target error data in the specified storage with the reduced data in the check table corresponding to the first check rule to obtain the same data as third data to be deleted when the data change condition indicates that the data in the check table corresponding to the first check rule is reduced, wherein the target error data in the specified storage is the error data in the check table corresponding to the first check rule recorded in the specified storage;
and deleting the third data to be deleted in the error data of the checking table corresponding to the first checking rule of the specified storage record.
13. The apparatus of claim 8, wherein the error data update sub-module is specifically configured to:
comparing the data added in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule is added, and obtaining wrong data serving as third data to be updated;
and adding the third data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record.
14. The apparatus of claim 8, wherein the error data update submodule is specifically configured to:
comparing the modified data in the check table corresponding to the first check rule with the full data in the reference table corresponding to the first check rule under the condition that the data change condition indicates that the data in the check table corresponding to the first check rule are modified, and obtaining wrong data serving as fourth data to be updated;
comparing the data before modification in the check table corresponding to the first check rule with the target error data in the designated storage to obtain the same data as fourth data to be deleted, wherein the target error data in the designated storage is the error data in the check table corresponding to the first check rule recorded in the designated storage;
and adding the fourth data to be updated in the error data of the check table corresponding to the first check rule of the specified storage record, and deleting the fourth data to be deleted.
15. An electronic device comprising a processor and a memory;
the memory is used for storing a computer program;
the processor is configured to implement the data checking method according to any one of claims 1 to 7 when executing the program stored in the memory.
16. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, implements the data checking method according to any one of claims 1 to 7.
CN202011021590.8A 2020-09-25 2020-09-25 Data checking method and device, electronic equipment and storage medium Active CN112148721B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011021590.8A CN112148721B (en) 2020-09-25 2020-09-25 Data checking method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011021590.8A CN112148721B (en) 2020-09-25 2020-09-25 Data checking method and device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112148721A CN112148721A (en) 2020-12-29
CN112148721B true CN112148721B (en) 2022-08-19

Family

ID=73897023

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011021590.8A Active CN112148721B (en) 2020-09-25 2020-09-25 Data checking method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112148721B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112988736B (en) * 2021-05-20 2021-08-03 睿至科技集团有限公司 Mass data quality checking method and system

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001337846A (en) * 2000-03-23 2001-12-07 Mitsubishi Electric Corp System and method for supporting quality inspection for software
JP2006190058A (en) * 2005-01-06 2006-07-20 Fujitsu Ten Ltd Abnormality detection device and method
CN101079749A (en) * 2007-06-27 2007-11-28 中国移动通信集团四川有限公司 Data consistency detection method
CN107870827A (en) * 2017-11-07 2018-04-03 中国银行股份有限公司 Data quality control method and device based on verification
CN110020550A (en) * 2019-02-21 2019-07-16 阿里巴巴集团控股有限公司 Check appraisal procedure, device and the equipment of platform
CN110704404A (en) * 2019-08-29 2020-01-17 苏宁云计算有限公司 Data quality checking method, device and system
CN110837496A (en) * 2019-11-08 2020-02-25 浪潮云信息技术有限公司 Data quality management method and system based on dynamic sql
CN111008241A (en) * 2019-11-14 2020-04-14 微民保险代理有限公司 Data synchronization method and device, storage medium and computer equipment

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001337846A (en) * 2000-03-23 2001-12-07 Mitsubishi Electric Corp System and method for supporting quality inspection for software
JP2006190058A (en) * 2005-01-06 2006-07-20 Fujitsu Ten Ltd Abnormality detection device and method
CN101079749A (en) * 2007-06-27 2007-11-28 中国移动通信集团四川有限公司 Data consistency detection method
CN107870827A (en) * 2017-11-07 2018-04-03 中国银行股份有限公司 Data quality control method and device based on verification
CN110020550A (en) * 2019-02-21 2019-07-16 阿里巴巴集团控股有限公司 Check appraisal procedure, device and the equipment of platform
CN110704404A (en) * 2019-08-29 2020-01-17 苏宁云计算有限公司 Data quality checking method, device and system
CN110837496A (en) * 2019-11-08 2020-02-25 浪潮云信息技术有限公司 Data quality management method and system based on dynamic sql
CN111008241A (en) * 2019-11-14 2020-04-14 微民保险代理有限公司 Data synchronization method and device, storage medium and computer equipment

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Big Data Quality:A Quality Dimensions Evaluation;Ikbal Taleb et al.;《IEEE Xplore》;20170116;全文 *
基于增量式FP-Growth算法的关联规则挖掘模型设计;高权 等;《信息技术与信息化》;20200331;全文 *
生化检验结果自动审核规则的制定及其应用;杨勇等;《临床检验杂志》;20160228(第02期);全文 *

Also Published As

Publication number Publication date
CN112148721A (en) 2020-12-29

Similar Documents

Publication Publication Date Title
CN107798047B (en) Repeated work order detection method, device, server and medium
CN111352902A (en) Log processing method and device, terminal equipment and storage medium
US11036699B2 (en) Method for computing distinct values in analytical databases
CN109783346B (en) Keyword-driven automatic testing method and device and terminal equipment
CN111475105A (en) Monitoring data storage method, device, server and storage medium
US11036701B2 (en) Data sampling in a storage system
CN112148721B (en) Data checking method and device, electronic equipment and storage medium
CN111435327B (en) Log record processing method, device and system
CN111221742A (en) Test case updating method and device, storage medium and server
CN110727895A (en) Sensitive word sending method and device, electronic equipment and storage medium
CN112948389B (en) MD 5-based database table data comparison method and device
CN112363944A (en) Method and equipment for comparing return values of multiple environment interfaces
CN114020813A (en) Data comparison method, device and equipment based on Hash algorithm and storage medium
CN111782661A (en) Data storage method, data query method and device
US7822721B2 (en) Correction server for large database systems
CN113392022B (en) Test requirement analysis method, device, computer readable medium and program product
CN113918593A (en) Method and device for identifying financial data abnormity, storage medium and computing equipment
CN114942927A (en) Database data comparison method and device
TWM640987U (en) video access system
CN117290227A (en) Verification method, device, equipment, medium and program product for nonstandard software product
CN117033360A (en) Bank data restoration method, device, equipment and medium based on blood margin analysis
CN114416735A (en) Data storage method, query method and system
CN112559331A (en) Test method and device
CN117667882A (en) Method and device for checking sql code
CN116126795A (en) Log retrieval method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant