CN112417841B

CN112417841B - Data verification method

Info

Publication number: CN112417841B
Application number: CN202011311055.6A
Authority: CN
Inventors: 仇越; 张庆晓; 张东文
Original assignee: Chaozhou Zhuoshu Big Data Industry Development Co Ltd
Current assignee: Chaozhou Zhuoshu Big Data Industry Development Co Ltd
Priority date: 2020-11-20
Filing date: 2020-11-20
Publication date: 2023-09-05
Anticipated expiration: 2040-11-20
Also published as: CN112417841A

Abstract

The invention relates to the field of data acquisition, and particularly provides a data verification method, which comprises the following steps: s1, constructing a result table of the same data acquisition object in a similar manner; s2, setting expected data quantity, calculating the data quantity in the current period, and comparing and calculating with the expected data quantity; s3, checking the data quantity of the current data acquisition object for nearly three times, and comparing the current data quantity with the historical data quantity to calculate a gap; s4, selecting important fields in a data table of the current data acquisition object, setting an expected null rate, and calculating the null rate; s5, comparing the null rate of the important fields in the twice-collected data table. Compared with the prior art, the data verification method is beneficial to ensuring that the quality of the acquisition result of each data acquisition object is basically consistent, can be basically consistent with the expected result, and provides good data quality guarantee for subsequent data mining.

Description

Data verification method

Technical Field

The invention relates to the field of data acquisition, and particularly provides a data verification method.

Background

With the advent of the big data age, the impact of data has gradually expanded, and more enterprises need valuable data to support their own business. The establishment of a data warehouse is a long-term process, and the same data acquisition object probably needs long-term continuous acquisition, but how to ensure the consistency of the quality and the expectation of each data acquisition is an important target of data acquisition.

Disclosure of Invention

The invention aims at the defects of the prior art and provides a data verification method with strong practicability.

The technical scheme adopted for solving the technical problems is as follows:

a data verification method, comprising the steps of:

s1, constructing a result table of the same data acquisition object in a similar manner;

s2, setting expected data quantity, calculating the data quantity in the current period, and comparing and calculating with the expected data quantity;

s3, checking the data quantity of the current data acquisition object for nearly three times, and comparing the current data quantity with the historical data quantity to calculate a gap;

s4, selecting important fields in a data table of the current data acquisition object, setting an expected null rate, and calculating the null rate;

s5, comparing the null rate of the important fields in the twice-collected data table.

Further, in step S1, the result table of the same data collection object is built with a fixed table structure and a fixed table name prefix, and time is used as a table name suffix, so that the result table is different except for the table time suffix, and other table basic information is the same.

Preferably, the results tables acquired at different times of the same data acquisition object must be identical in structure.

Further, in step S2, the latest data amount of the data acquisition object is counted and a difference is made between the latest data amount and the expected amount, a difference rate is calculated, then whether the latest data amount is within the range of the expected difference rate is checked, and if not, the data acquisition object is subjected to the complementary acquisition processing.

Further, in step S3, the data amount of the last three collection result tables of the data collection object is calculated and compared with the data amount of the latest collection result table;

alternatively, the total four data volumes are plotted as a line graph to view volatility;

if the gap is too large or the volatility is not within expectations, the data acquisition object is analyzed and processed again.

Further, in step S4, selecting an important field in the acquisition result table of the current data acquisition object, calculating the duty ratio of the number of records with empty selected field content to the total data, namely the null value rate, and comparing with the expected null value rate to see whether the number of records is within the range of the expected null value rate;

if the data is not in the range, searching whether the structuring failure problem occurs in the data acquisition process.

Records in the results table for each acquisition regarding the amount of data and the null rate of key fields must exist and be queriable.

Preferably, records in the results table for each acquisition must exist and be queriable with respect to the amount of data and the null rate of the key fields.

Further, in step S5, on the premise of step S4, the null value rate of the same field in the same result table of the same data acquisition object is checked, and a difference value is calculated with the null value rate in the present period, if the difference value is not in the expected range, the data acquisition personnel needs to process in time.

Compared with the prior art, the data verification method has the following outstanding beneficial effects:

the invention is beneficial to ensuring the quality of the acquisition result of each data acquisition object to be basically consistent with the expected result, and provides good data quality guarantee for the subsequent data mining.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a data verification method.

Description of the embodiments

In order to provide a better understanding of the aspects of the present invention, the present invention will be described in further detail with reference to specific embodiments. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

A preferred embodiment is given below:

as shown in fig. 1, a data verification method in this embodiment includes the following steps:

s1, constructing a result table of the same data acquisition object in a similar manner:

and establishing a result table of the same data acquisition object in a fixed table structure, and establishing a fixed table name prefix and taking time as a table name suffix as an auxiliary, so that the basic information of other tables except for the table time suffix is the same in each result table. And precondition guarantee is provided for subsequent verification.

The result tables acquired by the same data acquisition object in different periods must be identical in structure.

S2, setting expected data volume, calculating the current data volume, and comparing the current data volume with the expected volume to calculate:

and counting the latest data quantity of the data acquisition object, calculating a difference value with the expected quantity, checking whether the difference value is within the range of the expected difference value, and if not, carrying out processing such as compensation acquisition on the data acquisition object.

S3, checking the data quantity of the current data acquisition object for nearly three times, and comparing the current data quantity with the historical data quantity to calculate a gap:

calculating and comparing the difference value between the data volume of the last three acquisition result tables of the data acquisition object and the data volume of the latest acquisition result table; or a total of four data volumes are plotted as a line graph to see volatility. If the gap is too large or the volatility is not within the expectations, the data acquisition object needs to be analyzed again for processing.

S4, selecting important fields in a data table of the current data acquisition object, setting an expected null rate, and calculating the null rate:

and selecting an important field in a collection result table of the current data collection object, calculating the duty ratio of the number of records with empty selected field content to the total data, namely the null value rate, comparing the duty ratio with the expected null value rate, and checking whether the duty ratio is in the range of the expected null value rate. If not, a search is needed to determine if a structuring failure problem occurs during data acquisition.

S5, comparing the null value rate of important fields in the twice-collected data table:

and on the premise of S4, checking the null value rate of the same field in the same result table of the same data acquisition object, calculating a difference value with the null value rate at the present period, and if the difference value is not in the expected range, timely processing by data acquisition personnel.

The above-mentioned specific embodiments are merely specific examples of the present invention, and the scope of the present invention is not limited to the specific embodiments, and any suitable changes or substitutions made by those skilled in the art, which conform to the technical solutions described in the claims of the present invention, should fall within the scope of the present invention.

Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made therein without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims

1. A data verification method, comprising the steps of:

s1, constructing a data table of the same data acquisition object in a similar manner;

the data table of the same data acquisition object is established by a fixed table structure and a fixed table name prefix, and time is taken as a table name suffix, so that the data table has the same basic information except for the table time suffix each time;

the data tables acquired by the same data acquisition object in different periods must have the same structure;

counting the data quantity of a data table which is acquired by a data acquisition object at the present time, making a difference value between the present data quantity and an expected quantity, calculating a difference value rate, checking whether the difference value rate is within the range of the expected difference value rate, and if not, carrying out supplementary acquisition processing on the data acquisition object;

s3, checking the data quantity of the current data acquisition object for three times in the past, and comparing the current data quantity with the historical data quantity to calculate a gap;

calculating and comparing the data volume of the data collection table of the data collection object for the past three times with the data volume of the data collection table at present;

if the gap is too large or the volatility is not within the expectations, analyzing and processing the data acquisition object again;

selecting an important field in a collection data table of a current data collection object, calculating the duty ratio of the number of records with empty selected field content to the total data, namely the null value rate, comparing the duty ratio with the expected null value rate, and checking whether the duty ratio is in the range of the expected null value rate;

if the data is not in the range, searching whether the structural failure problem occurs in the data acquisition process;

s5, comparing the null rate of important fields in the twice-collected data table;

on the premise of step S4, checking the null value rate of the same field in the same data table of the same data acquisition object, calculating a difference value with the current null value rate, and if the difference value is not in the expected range, timely processing by data acquisition personnel.