CN110377454B - Data verification method and device, computer equipment and storage medium - Google Patents
Data verification method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110377454B CN110377454B CN201910521896.0A CN201910521896A CN110377454B CN 110377454 B CN110377454 B CN 110377454B CN 201910521896 A CN201910521896 A CN 201910521896A CN 110377454 B CN110377454 B CN 110377454B
- Authority
- CN
- China
- Prior art keywords
- data
- field
- verified
- original
- subfield
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1004—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
Abstract
The application relates to a data verification method, a data verification device, computer equipment and a storage medium. The method relates to a data consistency testing technology, and comprises the following steps: acquiring data to be verified and original data corresponding to the data to be verified; determining a field to be verified in the data to be verified and an original field in the original data according to a preset data verification rule; comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent; according to the data splitting number, splitting the field to be verified to obtain a subfield to be verified, and splitting the original field to obtain an original subfield; and comparing the sub-field hash value of the sub-field to be verified with the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result. By adopting the method, the data verification efficiency can be improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data verification method and apparatus, a computer device, and a storage medium.
Background
With the development of computer technology and the coming of cloud era, the data volume stored and processed by each business system increases in a geometric progression, and big data attracts more and more attention. During the processing processes of data transmission, data migration or system upgrade, the data before and after processing needs to be checked, so that the integrity and consistency of the data are ensured.
However, with the rapid increase of the data volume, the data verification efficiency is low by the current data verification method of data-to-data comparison.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data verification method, an apparatus, a computer device, and a storage medium, which can improve data verification efficiency.
A method of data verification, the method comprising:
acquiring data to be verified and original data corresponding to the data to be verified;
determining a field to be checked in the data to be checked and an original field in the original data according to a preset data checking rule;
comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent;
according to the data splitting number, splitting the field to be verified to obtain subfields to be verified, and splitting the original field to obtain an original subfield;
and comparing the sub-field hash value of the sub-field to be checked with the sub-field hash value of the original sub-field, and obtaining a data checking result according to the sub-field comparison result.
In one embodiment, the obtaining of the data to be verified and the original data corresponding to the data to be verified includes:
determining the service type of the data to be checked;
inquiring a preset data check priority corresponding to the service type;
and acquiring the data to be verified and the original data corresponding to the data to be verified according to the data verification priority.
In one embodiment, determining, according to a preset data verification rule, a field to be verified in the data to be verified and an original field in the original data includes:
inquiring a preset data verification rule, and determining a data verification section according to the data verification rule;
and according to the data verification section, respectively extracting fields to be verified from the data to be verified, and extracting original fields from the original data.
In one embodiment, comparing the field hash value of the field to be checked with the field hash value of the original field includes:
acquiring a preset field hash function;
respectively calculating a field hash value of a field to be checked and a field hash value of an original field according to a field hash function;
and comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result.
In one embodiment, when the field comparison results are inconsistent, determining the data splitting number corresponding to the data to be checked according to the data length of the data to be checked includes:
when the field comparison result is inconsistent, determining the data length of the data to be checked;
acquiring a preset check data division rule;
and determining the data splitting number of the data to be verified according to the data length and the verification data division rule.
In one embodiment, comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield, and obtaining the data verification result according to the subfield comparison result includes:
acquiring a preset subfield hash function;
respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the subfield hash function;
comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result;
and obtaining error data segments according to the inconsistent subfields to be verified in the subfield comparison results, wherein the data verification results comprise the error data segments.
In one embodiment, after obtaining the error data segment according to the inconsistent sub-fields to be checked in the sub-field comparison result, the method further includes:
acquiring an original data segment corresponding to the error data segment;
and carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
A data verification apparatus, the apparatus comprising:
the data acquisition module is used for acquiring data to be verified and original data corresponding to the data to be verified;
the verification field determining module is used for determining a field to be verified in the data to be verified and an original field in the original data according to a preset data verification rule;
the field comparison module is used for comparing the field hash value of the field to be checked with the field hash value of the original field, and when the field comparison result is inconsistent, determining the data splitting number corresponding to the data to be checked according to the data length of the data to be checked;
the field splitting module is used for splitting the field to be verified respectively to obtain a subfield to be verified and splitting the original field to obtain an original subfield according to the data splitting number;
and the verification result acquisition module is used for comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield and obtaining a data verification result according to the subfield comparison result.
A computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:
acquiring data to be verified and original data corresponding to the data to be verified;
determining a field to be verified in the data to be verified and an original field in the original data according to a preset data verification rule;
comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent;
according to the data splitting number, splitting the field to be verified to obtain a subfield to be verified, and splitting the original field to obtain an original subfield;
and comparing the sub-field hash value of the sub-field to be verified with the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result.
A computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring data to be verified and original data corresponding to the data to be verified;
determining a field to be checked in the data to be checked and an original field in the original data according to a preset data checking rule;
comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent;
according to the data splitting number, splitting the field to be verified to obtain a subfield to be verified, and splitting the original field to obtain an original subfield;
and comparing the sub-field hash value of the sub-field to be verified with the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result.
According to the data checking method, the data checking device, the computer equipment and the storage medium, compared data quantity during data checking is reduced by comparing the field hash value of the field to be checked in the data to be checked with the field hash value of the original field in the original data, when the field comparison result is inconsistent, the field to be checked and the original field are split according to the data length of the data to be checked, and error data can be quickly positioned by comparing the subfield hash value of the subfield to be checked with the subfield hash value of the original subfield, so that the data checking efficiency is improved.
Drawings
FIG. 1 is a diagram illustrating an exemplary implementation of a data verification method;
FIG. 2 is a flow diagram illustrating a data verification method according to one embodiment;
FIG. 3 is a flow diagram illustrating field checking in one embodiment;
FIG. 4 is a flow chart illustrating a data verification method according to another embodiment;
FIG. 5 is a block diagram showing the structure of a data verification apparatus according to an embodiment;
FIG. 6 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
The data verification method provided by the application can be applied to the application environment shown in fig. 1. The server 102 is connected to a verification database server 104 and an original database server 106, respectively. The server 102 obtains data to be verified from the verification database server 104, obtains original data corresponding to the data to be verified from the original database server 106, splits the field to be verified and the original field according to the data length of the data to be verified when the field hash value of the field to be verified and the field hash value of the original field in the original data are not consistent, and can quickly locate error data to obtain a data verification result by comparing the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield. The server 102, the verification database server 104 and the original database server 106 may be implemented by a single server or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a data verification method is provided, which is described by taking the method as an example applied to the server 102 in fig. 1, and includes the following steps:
step S201: and acquiring the data to be verified and the original data corresponding to the data to be verified.
The data to be verified is data that needs to be verified, and may be data after data migration, data after system upgrade, or data received by a receiver after transmission, for example. The original data is source data corresponding to the data to be verified, and may be used as reference data when the data to be verified is subjected to data verification processing, that is, in the data migration, system upgrade, or data transmission process, the integrity and consistency of the obtained data need to be ensured. In a specific data migration application, data to be verified can be acquired from a database after data migration, and original data corresponding to the data to be verified can be acquired from a database before data migration.
Step S203: and determining a field to be checked in the data to be checked and an original field in the original data according to a preset data checking rule.
The data verification rule is used for determining a data field needing to be verified, and is preset according to system differences to which the data to be verified and the original data belong respectively. For example, for data to be verified and original data in data migration, service system platforms to which the data respectively belong before and after the data migration are different, and a data format may have corresponding conversion processing; furthermore, it may not be necessary to check for certain fields in the data. When data verification is performed, data differences caused by differences of service system platforms need to be eliminated, and data fields which do not need to be verified are filtered, that is, the data fields which need to be verified are determined according to the data verification rules. Specifically, a preset data verification rule may be queried, a field to be verified is determined from the data to be verified according to the data verification rule, and an original field is determined from the original data, where the field to be verified corresponds to the original field, and the original field may be used as a reference field of the field to be verified during verification processing.
Step S205: and comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent.
The field hash value of the field to be checked and the field hash value of the original field may be calculated by a preset field hash Algorithm, such as an MD5Message Digest Algorithm (MD 5Message-Digest Algorithm). The data splitting number is the number of the sub-fields to be verified after the fields to be verified are split, the data splitting number can be correspondingly set according to the length of the fields to be verified, a fixed data splitting number can also be set, a fixed data splitting length can also be set, and the corresponding data splitting number is determined according to the data splitting length and the data to be verified. Specifically, after a field to be verified in the data to be verified and an original field in the original data are obtained, a field hash value of the field to be verified and a field hash value of the original field are respectively obtained through a preset hash algorithm, the field hash values and the original field hash values are compared, a field verification result can be obtained according to the field comparison result, and the data verification result comprises the field verification result. By comparing the field hash values, the data volume for comparison can be effectively reduced, and the data verification efficiency is improved.
Further, when the field comparison result is the comparison consistency, it is indicated that the field to be checked is identical to the original field, that is, the original data corresponding to the data to be checked is identical, the obtained field check result is a check pass, and the data check result includes a field check result, thereby completing the data check. And when the field comparison result is inconsistent, the field to be verified is different from the original field, namely the original data corresponding to the data to be verified is different, and the field verification result indicates that the verification is failed. At this moment, the error data which cannot be verified is positioned in an error mode, specifically, the data splitting number corresponding to the data to be verified is determined according to the data length of the data to be verified, the field to be verified and the original field are split according to the data splitting number, and the split sub-field to be verified and the split sub-field are compared, so that the error data can be accurately positioned.
Step S207: and according to the data splitting number, splitting the field to be verified to obtain the subfield to be verified, and splitting the original field to obtain the original subfield.
When the field comparison result is inconsistent, the data to be verified is different from the original data, and the error data can be positioned by splitting the field to be verified into the subfields to be verified. Specifically, after the data splitting number is obtained, the field to be verified and the original field are respectively split according to the data splitting number, and the subfield to be verified and the original subfield are correspondingly obtained. Each subfield to be verified and each original subfield correspond to each other, and the original subfield can be used as reference data in the verification processing of the subfield to be verified, that is, the verification of the subfield to be verified can be realized by comparing the subfield to be verified and the original subfield.
Step S209: and comparing the sub-field hash value of the sub-field to be verified with the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result.
After the subfield to be verified and the original subfield are obtained, calculating a subfield hash value of the subfield to be verified and a subfield hash value of the original subfield, which can be specifically calculated according to a preset subfield hash algorithm. And comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result, and obtaining a data verification result according to the subfield comparison result, thereby realizing the quick verification of the data to be verified and accurately positioning error data. By comparing the subfield hash value of each subfield to be verified with the subfield hash value of the original subfield, the subfield to be verified in the data to be verified with inconsistent subfield comparison results can be determined, so that the location of error data in the data to be verified is realized, and a data verification result is further obtained, wherein the data verification result can include the error data which does not pass the data verification and corresponds to the error data.
According to the data verification method, the field hash value of the field to be verified in the data to be verified and the field hash value of the original field in the original data are compared, the compared data volume during data verification is reduced, when the field comparison results are inconsistent, the field to be verified and the original field are split according to the data length of the data to be verified, the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield are compared, the error data can be quickly positioned, and the data verification efficiency is improved.
In one embodiment, the obtaining of the data to be verified and the original data corresponding to the data to be verified includes: determining the service type of the data to be checked; inquiring a preset data check priority corresponding to the service type; and acquiring the data to be verified and the original data corresponding to the data to be verified according to the data verification priority.
The original data is source data corresponding to the data to be verified, and is used as reference data when the data to be verified is subjected to data verification processing, that is, the data to be verified is compared with the corresponding original data, so that the data verification processing of the data to be verified is realized. For a business system with different business rules, the relation among various data in a business system database is different, the business system database has different data levels, and different data verification requirements are met correspondingly, for example, some data are combined data which are further obtained on the basis of other data according to certain business rules. In addition, different data in the database have different access frequencies, and also correspond to different data verification requirements. Therefore, during data verification processing, data can be verified in sequence by combining data verification requirements of various data, so that normal operation and use of the data are ensured.
Specifically, when the data to be verified and the original data corresponding to the data to be verified are obtained, the service type to which the data to be verified belong is determined, and the service type can be determined according to the service system to which the data to be verified belong. And querying a preset data verification priority corresponding to the service type, wherein the data verification priority reflects the data verification priority requirement of the data to be verified of the service type, and the data to be verified under different service types correspond to different data verification priorities. The data verification priority can be determined by performing priority division on various data according to the service rule and historical access data of the service system. And acquiring the data to be verified and the original data corresponding to the data to be verified according to the determined data verification priority, wherein the data to be verified and the original data corresponding to the data to be verified can be extracted from a verification database server and an original database server. The data to be checked is obtained in sequence for checking according to the preset data checking priority corresponding to the data to be checked, so that the data with high data level or high access frequency can be guaranteed to be checked preferentially, and normal access and use of the data are guaranteed.
In one embodiment, determining the field to be checked in the data to be checked and the original field in the original data according to a preset data checking rule includes: inquiring a preset data verification rule, and determining a data verification section according to the data verification rule; and according to the data verification section, respectively extracting fields to be verified from the data to be verified, and extracting original fields from the original data.
The data verification rule is used for determining a data field needing to be verified, and is preset according to system differences to which the data to be verified and the original data respectively belong. In this embodiment, when determining the field to be verified in the data to be verified and the original field in the original data according to the data verification rule, a preset data verification rule is queried, and a data verification segment is determined according to the data verification rule, where the data verification segment is a data segment that needs to be subjected to data verification processing. And according to the data verification segment, extracting fields to be verified from the data to be verified respectively, and extracting original fields from the original data. The field to be checked corresponds to the original field, and the original field can be used as a reference field of the field to be checked during checking processing.
In one embodiment, as shown in fig. 3, the step of checking the field, i.e. comparing the field hash value of the field to be checked with the field hash value of the original field, includes:
step S301: and acquiring a preset field hash function.
In the processes of data migration, system upgrade, data transmission and the like, the data are processed in the form of 0 and 1 coded data of bit positions, the length of a bit code corresponding to each data is very long, if the data are directly compared one by one according to the bit codes during data verification, the data verification efficiency is extremely low, and errors are easy to occur. By mapping and compressing the data into short parameters and then checking, the data checking processing efficiency can be effectively improved, for example, the data can be compressed and mapped into a short message digest based on a hash algorithm.
Specifically, the field hash function may compress and map the input fields to be checked and the original fields into a corresponding fixed-length message digest, that is, a corresponding field hash value. The length of the field hash value is far smaller than that of the field to be checked, and then field checking is carried out based on the field hash value, so that the data volume for comparison can be effectively reduced, and the data checking processing efficiency is improved. In this embodiment, when performing field verification on a field to be verified, a preset field hash function is obtained, for example, the field hash function is constructed based on MD4, MD5, and SHA 1.
Step S303: and respectively calculating the field hash value of the field to be checked and the field hash value of the original field according to the field hash function.
And after the field hash function is obtained, respectively calculating the field hash value of the field to be checked and the field hash value of the original field according to the field hash function. Specifically, the field to be verified and the original field may be respectively input into a field hash function, so as to obtain a field hash value of the field to be verified and a field hash value of the original field.
Step S305: and comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result.
And after the field hash value of the field to be checked and the field hash value of the original field are obtained, comparing the field hash values to obtain a field comparison result. Specifically, when the field hash value of the field to be verified is the same as the field hash value of the original field, it is indicated that the field to be verified is the same as the original field, that is, the original data corresponding to the data to be verified is the same, the obtained field verification result is a verification pass, and the data verification result includes the field verification result, thereby completing the data verification. And when the field comparison result is inconsistent, the field to be verified is different from the original field, namely the original data corresponding to the data to be verified is different, and the field verification result indicates that the verification is failed. At this time, the error data may be further located to obtain an error data segment, and the data verification result may include the error data segment.
In one embodiment, when the field comparison results are inconsistent, determining the data splitting number corresponding to the data to be checked according to the data length of the data to be checked includes: when the field comparison result is inconsistent, determining the data length of the data to be checked; acquiring a preset check data division rule; and determining the data splitting number of the data to be verified according to the data length and the verification data division rule.
After the field hash value of the field to be verified and the field hash value of the original field are compared to obtain a field comparison result, if the field comparison result is inconsistent, the field to be verified and the original field are different, namely the original data corresponding to the data to be verified are different, and the field verification result indicates that the data to be verified does not pass the verification.
Specifically, when the data splitting number of the field to be checked and the original field is determined, the data length of the data to be checked is determined. The data to be verified may be analyzed to determine the data length of the data to be verified, for example, the length of the data to be verified may be calculated based on the size of the data to be verified. And then, acquiring a preset check data division rule, wherein the check data division rule can comprise data splitting numbers corresponding to various data lengths, namely different data lengths correspond to different data splitting numbers. The check data dividing rule can be preset according to actual processing resources so as to divide the data to be checked to obtain data segments with proper data length. For example, the sub-data length of the sub-data obtained after the data splitting is performed may be determined, that is, the sub-field length of the sub-field to be verified is determined, and the data splitting number may be obtained by combining the data length of the field to be verified; or, a splitting ratio, such as 20%, 10%, etc., may be preset, and the field to be checked is split according to the splitting ratio, so as to obtain the corresponding data splitting number.
And determining the data length of the data to be verified, obtaining a verification data division rule, and determining the data splitting number of the data to be verified according to the verification data division rule and the data length. And splitting the field to be verified and the original field according to the data splitting number to correspondingly obtain the subfield to be verified and the original subfield, and positioning error data according to the subfield to be verified and the original subfield.
In one embodiment, comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield, and obtaining the data verification result according to the subfield comparison result includes: acquiring a preset subfield hash function; respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the subfield hash function; comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result; and obtaining error data segments according to the inconsistent subfields to be verified in the subfield comparison results, wherein the data verification results comprise the error data segments.
When error data is positioned, the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield can be compared, and a data verification result is obtained according to the subfield comparison result. Specifically, a preset subfield hash function is obtained, the subfield hash function corresponds to the field hash function, and subfield hash values of the subfield to be verified and the original subfield can be obtained through the subfield hash function respectively. In a specific application, the subfield hash function and the field hash function may be the same or different. And respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the obtained subfield hash function, and performing subfield comparison, namely comparing the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield to obtain a subfield comparison result.
Considering that the field comparison results are inconsistent, at least one of the subfield comparison results is inconsistent, and the subfield to be verified corresponding to the inconsistent subfield comparison results is the error data field, so that the error data can be positioned. Specifically, the sub-field to be checked corresponding to the sub-field comparison result which is inconsistent is determined, and is used as the error data segment, so as to obtain the data check result, which may include the error data segment. By comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield, and according to the inconsistent subfield to be verified in the subfield comparison result, the error data can be accurately positioned, and the data verification efficiency is improved.
In one embodiment, after obtaining the error data segment according to the inconsistent sub-fields to be checked in the sub-field comparison result, the method further includes: acquiring an original data segment corresponding to the error data segment; and carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
After the error data is positioned to obtain the error data segment, bit check can be further performed on the error data to determine the specific error bit. Specifically, after the error data segment is obtained, the original data segment corresponding to the error data segment is obtained, and the original data segment may be determined from the original sub-field. The bit check is carried out on the error data segment and the original data segment, specifically, the bit of the error data segment and the bit of the original data segment can be compared one by one to obtain a bit check result, and the error bit in the error data segment can be accurately determined through the bit check result. Compared with the data to be checked and the field to be checked, the data length of the error data segment and the original data segment is shorter, the bit positions of the error data segment and the original data segment are compared one by one, the checking processing efficiency can be ensured, and the accurate positioning of the error bit positions can be realized.
In one embodiment, as shown in fig. 4, there is provided a data verification method, including:
step S401: determining the service type of the data to be checked;
step S402: inquiring a preset data check priority corresponding to the service type;
step S403: and acquiring the data to be verified and the original data corresponding to the data to be verified according to the data verification priority.
And the original data is the source data corresponding to the data to be verified, and the data to be verified is compared with the corresponding original data so as to realize data verification processing of the data to be verified. In this embodiment, according to the determined data verification priority, the data to be verified and the original data corresponding to the data to be verified are obtained, and are specifically extracted from the verification database server and the original database server.
Step S404: inquiring a preset data verification rule, and determining a data verification section according to the data verification rule;
step S405: and according to the data verification section, respectively extracting fields to be verified from the data to be verified, and extracting original fields from the original data.
The data verification rule is used for determining a data field needing to be verified, and is preset according to system differences to which the data to be verified and the original data respectively belong. And extracting fields to be checked from the data to be checked respectively according to the data checking section driven by the data checking rule, and extracting the original fields from the original data. The field to be checked corresponds to the original field, and the original field can be used as a reference field of the field to be checked during checking processing.
Step S406: comparing the field hash value of the field to be checked with the field hash value of the original field;
step S407: when the field comparison results are inconsistent, determining the data length of the data to be checked;
step S408: acquiring a preset check data division rule;
step S409: determining the data splitting number of the data to be verified according to the data length and the verification data division rule;
step S410: and according to the data splitting number, splitting the field to be verified to obtain the subfield to be verified, and splitting the original field to obtain the original subfield.
In this embodiment, comparing the field hash value includes: acquiring a preset field hash function; respectively calculating a field hash value of a field to be checked and a field hash value of an original field according to a field hash function; and comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result. Wherein the field hash function is MD5.
When the field comparison result is inconsistent, it indicates that the field to be verified is different from the original field, that is, the original data corresponding to the data to be verified is different, and the field verification result indicates that the verification fails. At this moment, the error data which is not verified can be positioned in an error mode, specifically, the data splitting number can be determined, the field to be verified and the original field are split according to the data splitting number, and the split sub field to be verified and the original sub field are compared, so that the error data can be accurately positioned.
Step S411: acquiring a preset subfield hash function;
step S412: respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the subfield hash function;
step S413: comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result;
step S414: and obtaining error data segments according to the inconsistent subfields to be verified in the subfield comparison results, wherein the data verification results comprise the error data segments.
And after the sub-field to be verified and the original sub-field are obtained, calculating and comparing the sub-field hash value of the sub-field to be verified and the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result, thereby realizing the quick verification of the data to be verified and accurately positioning error data.
Step S415: and carrying out bit check on the error data segment to obtain a bit check result, wherein the data check result comprises a bit check result.
After the error data is positioned to obtain the error data segment, bit check can be further performed on the error data to determine the specific error bit. Specifically, the bit check of the erroneous data segment includes: acquiring an original data segment corresponding to the error data segment; and carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
It should be understood that although the various steps in the flow charts of fig. 2-4 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-4 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 5, there is provided a data verification apparatus including: a data obtaining module 501, a check field determining module 503, a field comparing module 505, a field comparing module 507, and a check result obtaining module 509, wherein:
a data obtaining module 501, configured to obtain data to be verified and original data corresponding to the data to be verified;
a check field determining module 503, configured to determine, according to a preset data check rule, a field to be checked in the data to be checked and an original field in the original data;
the field comparison module 505 is configured to compare a field hash value of a field to be verified with a field hash value of an original field, and determine a data splitting number corresponding to the data to be verified according to a data length of the data to be verified when the field comparison results are inconsistent;
a field comparison module 507, configured to split the field to be verified to obtain subfields to be verified and split the original field to obtain an original subfield according to the number of data splits;
and a verification result obtaining module 509, configured to compare the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield, and obtain a data verification result according to the subfield comparison result.
In one embodiment, the data obtaining module 501 includes a service type determining unit, a priority determining unit and a data obtaining unit, wherein: the service type determining unit is used for determining the service type of the data to be checked; the priority determining unit is used for inquiring the preset data checking priority corresponding to the service type; and the data acquisition unit is used for acquiring the data to be checked and the original data corresponding to the data to be checked according to the data checking priority.
In one embodiment, the check field determination module 503 includes a check segment determination unit and a check field extraction unit, wherein: the verification section determining unit is used for inquiring a preset data verification rule and determining a data verification section according to the data verification rule; and the check field extracting unit is used for respectively extracting the fields to be checked from the data to be checked according to the data check segment and extracting the original fields from the original data.
In one embodiment, the field comparison module 505 includes a field hash function unit, a field hash processing unit, and a field comparison unit, wherein: the field hash function unit is used for acquiring a preset field hash function; the field hash processing unit is used for respectively calculating the field hash value of the field to be checked and the field hash value of the original field according to a field hash function; and the field comparison unit is used for comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result.
In one embodiment, the field comparison module 505 comprises a data length determination unit, a partition rule acquisition unit, and a data splitting unit, wherein: the data length determining unit is used for determining the data length of the data to be checked when the field comparison result is inconsistent; the device comprises a division rule obtaining unit, a check data dividing unit and a verification unit, wherein the division rule obtaining unit is used for obtaining a preset check data division rule; and the data splitting unit is used for determining the data splitting number of the data to be verified according to the data length and the verification data division rule.
In one embodiment, the verification result obtaining module 509 includes a sub-field hash function unit, a sub-field hash processing unit, a sub-field comparison unit, and an error data positioning unit, wherein: the subfield hash function unit is used for acquiring a preset subfield hash function; the subfield hash processing unit is used for respectively calculating the subfield hash value of the subfield to be checked and the subfield hash value of the original subfield according to the subfield hash function; the subfield comparison unit is used for comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result; and the error data positioning unit is used for obtaining error data segments according to the inconsistent subfields to be verified in the subfield comparison results, and the data verification results comprise the error data segments.
In one embodiment, the system further comprises an original data segment acquisition module and a bit check module, wherein: the original data segment acquisition module is used for acquiring an original data segment corresponding to the error data segment; and the bit check module is used for carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
For specific limitations of the data verification apparatus, reference may be made to the above limitations of the data verification method, which is not described herein again. All or part of each module in the data checking device can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 6. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data verification method.
Those skilled in the art will appreciate that the architecture shown in fig. 6 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, there is provided a computer device comprising a memory storing a computer program and a processor implementing the following steps when the processor executes the computer program:
acquiring data to be verified and original data corresponding to the data to be verified;
determining a field to be checked in the data to be checked and an original field in the original data according to a preset data checking rule;
comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent;
according to the data splitting number, splitting the field to be verified to obtain subfields to be verified, and splitting the original field to obtain an original subfield;
and comparing the sub-field hash value of the sub-field to be verified with the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: determining the service type of the data to be checked; inquiring a preset data check priority corresponding to the service type; and acquiring the data to be verified and the original data corresponding to the data to be verified according to the data verification priority.
In one embodiment, the processor, when executing the computer program, further performs the steps of: inquiring a preset data verification rule, and determining a data verification section according to the data verification rule; and according to the data verification section, respectively extracting fields to be verified from the data to be verified, and extracting original fields from the original data.
In one embodiment, the processor, when executing the computer program, further performs the steps of: acquiring a preset field hash function; respectively calculating a field hash value of a field to be checked and a field hash value of an original field according to a field hash function; and comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: when the field comparison result is inconsistent, determining the data length of the data to be checked; acquiring a preset check data division rule; and determining the data splitting number of the data to be verified according to the data length and the verification data division rule.
In one embodiment, the processor, when executing the computer program, further performs the steps of: acquiring a preset subfield hash function; respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the subfield hash function; comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result; and obtaining error data segments according to the inconsistent subfields to be verified in the subfield comparison results, wherein the data verification results comprise the error data segments.
In one embodiment, the processor, when executing the computer program, further performs the steps of: acquiring an original data segment corresponding to the error data segment; and carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, performs the steps of:
acquiring data to be checked and original data corresponding to the data to be checked;
determining a field to be verified in the data to be verified and an original field in the original data according to a preset data verification rule;
comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent;
according to the data splitting number, splitting the field to be verified to obtain subfields to be verified, and splitting the original field to obtain an original subfield;
and comparing the sub-field hash value of the sub-field to be verified with the sub-field hash value of the original sub-field, and obtaining a data verification result according to the sub-field comparison result.
In one embodiment, the computer program when executed by the processor further performs the steps of: determining the service type of the data to be checked; inquiring a preset data check priority corresponding to the service type; and acquiring the data to be verified and the original data corresponding to the data to be verified according to the data verification priority.
In one embodiment, the computer program when executed by the processor further performs the steps of: inquiring a preset data verification rule, and determining a data verification section according to the data verification rule; and according to the data verification section, respectively extracting fields to be verified from the data to be verified, and extracting original fields from the original data.
In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring a preset field hash function; respectively calculating a field hash value of a field to be checked and a field hash value of an original field according to a field hash function; and comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result.
In one embodiment, the computer program when executed by the processor further performs the steps of: when the field comparison result is inconsistent, determining the data length of the data to be checked; acquiring a preset check data division rule; and determining the data splitting number of the data to be verified according to the data length and the verification data division rule.
In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring a preset subfield hash function; respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the subfield hash function; comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result; and obtaining error data segments according to the inconsistent subfields to be verified in the subfield comparison results, wherein the data verification results comprise the error data segments.
In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring an original data segment corresponding to the error data segment; and carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is specific and detailed, but not to be understood as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A method of data verification, the method comprising:
acquiring data to be verified and original data corresponding to the data to be verified;
determining a field to be checked in the data to be checked and an original field in the original data according to a preset data checking rule, wherein the data checking rule is set based on system differences to which the data to be checked and the original data belong respectively;
comparing the field hash value of the field to be verified with the field hash value of the original field, and determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified when the field comparison results are inconsistent;
according to the data splitting number, splitting the field to be verified to obtain subfields to be verified, and splitting the original field to obtain an original subfield;
and comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield, and obtaining a data verification result according to the subfield comparison result.
2. The method according to claim 1, wherein the obtaining of the data to be verified and the original data corresponding to the data to be verified comprises:
determining the service type of the data to be checked;
inquiring a preset data verification priority corresponding to the service type;
and acquiring the data to be verified and the original data corresponding to the data to be verified according to the data verification priority.
3. The method according to claim 1, wherein the determining, according to a preset data verification rule, a field to be verified in the data to be verified and an original field in the original data comprises:
inquiring a preset data verification rule, and determining a data verification section according to the data verification rule;
and according to the data verification segment, respectively extracting fields to be verified from the data to be verified, and extracting original fields from the original data.
4. The method of claim 1, wherein comparing the field hash value of the field to be checked with the field hash value of the original field comprises:
acquiring a preset field hash function;
respectively calculating the field hash value of the field to be checked and the field hash value of the original field according to the field hash function;
and comparing the field hash value of the field to be checked with the field hash value of the original field to obtain a field comparison result.
5. The method according to claim 4, wherein when the field comparison result is inconsistent, determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified comprises:
when the field comparison result is inconsistent, determining the data length of the data to be checked;
acquiring a preset check data division rule;
and determining the data splitting number of the data to be verified according to the data length and the verification data division rule.
6. The method according to any one of claims 1 to 5, wherein the comparing the sub-field hash value of the sub-field to be checked with the sub-field hash value of the original sub-field, and obtaining the data checking result according to the sub-field comparison result comprises:
acquiring a preset subfield hash function;
respectively calculating the subfield hash value of the subfield to be verified and the subfield hash value of the original subfield according to the subfield hash function;
comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield to obtain a subfield comparison result;
and obtaining an error data segment according to the inconsistent subfields to be checked in the subfield comparison results, wherein the data check results comprise the error data segment.
7. The method according to claim 6, further comprising, after obtaining the erroneous data segment according to the inconsistent sub-fields to be checked in the sub-field comparison result:
acquiring an original data segment corresponding to the error data segment;
and carrying out bit check on the error data segment and the original data segment to obtain a bit check result.
8. A data verification apparatus, the apparatus comprising:
the data acquisition module is used for acquiring data to be verified and original data corresponding to the data to be verified;
the verification field determining module is used for determining a field to be verified in the data to be verified and an original field in the original data according to a preset data verification rule, and the data verification rule is set based on system differences to which the data to be verified and the original data belong respectively;
the field comparison module is used for comparing the field hash value of the field to be verified with the field hash value of the original field, and when the field comparison result is inconsistent, determining the data splitting number corresponding to the data to be verified according to the data length of the data to be verified;
the field splitting module is used for splitting the field to be verified respectively according to the data splitting number to obtain a subfield to be verified and splitting the original field to obtain an original subfield;
and the verification result acquisition module is used for comparing the subfield hash value of the subfield to be verified with the subfield hash value of the original subfield and obtaining a data verification result according to the subfield comparison result.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910521896.0A CN110377454B (en) | 2019-06-17 | 2019-06-17 | Data verification method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910521896.0A CN110377454B (en) | 2019-06-17 | 2019-06-17 | Data verification method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110377454A CN110377454A (en) | 2019-10-25 |
| CN110377454B true CN110377454B (en) | 2023-02-07 |
Family
ID=68248985
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910521896.0A Active CN110377454B (en) | 2019-06-17 | 2019-06-17 | Data verification method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110377454B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110781134A (en) * | 2019-10-30 | 2020-02-11 | 泰康保险集团股份有限公司 | Data processing method and device, computer storage medium and electronic equipment |
| CN111046043A (en) * | 2019-12-11 | 2020-04-21 | 北京西骏数据科技股份有限公司 | Method for quickly and accurately checking database table |
| CN111339508A (en) * | 2020-02-28 | 2020-06-26 | 北京达佳互联信息技术有限公司 | Shared password analysis method and device, electronic equipment and storage medium |
| CN112559547B (en) * | 2020-12-24 | 2023-09-19 | 北京百度网讯科技有限公司 | Method and device for determining consistency among multiple storage object copies |
| CN113672634A (en) * | 2021-08-30 | 2021-11-19 | 中国平安人寿保险股份有限公司 | Policy data verification method, device, equipment and storage medium |
| CN114567485A (en) * | 2022-03-01 | 2022-05-31 | 北京全路通信信号研究设计院集团有限公司 | Data processing method, device, equipment and storage medium |
| CN115221245B (en) * | 2022-07-14 | 2023-07-14 | 广州纳指数据智能科技有限公司 | Intelligent data acquisition synchronization method, system and equipment |
| CN115760006B (en) * | 2022-11-29 | 2023-07-18 | 联仁健康医疗大数据科技股份有限公司 | Data correction method, device, electronic equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101702639B (en) * | 2009-11-23 | 2012-12-19 | 成都市华为赛门铁克科技有限公司 | Check value calculation method and device of cyclic redundancy check |
| CN102325167A (en) * | 2011-07-21 | 2012-01-18 | 杭州微元科技有限公司 | Verifying method for network file transmission |
| CN104581431B (en) * | 2014-11-28 | 2018-01-30 | 精宸智云（武汉）科技有限公司 | Video authentication method and device |
| CN106776105B (en) * | 2016-11-15 | 2020-02-21 | 惠州Tcl移动通信有限公司 | System startup file checking and compiling method |
-
2019
- 2019-06-17 CN CN201910521896.0A patent/CN110377454B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN110377454A (en) | 2019-10-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110377454B (en) | Data verification method and device, computer equipment and storage medium | |
| CN110209652B (en) | Data table migration method, device, computer equipment and storage medium | |
| CN109062788B (en) | Test case generation method and device, computer equipment and storage medium | |
| CN109040191B (en) | File downloading method and device, computer equipment and storage medium | |
| CN112559365A (en) | Test case screening method and device, computer equipment and storage medium | |
| CN112559364A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN110968647A (en) | Data storage method and device, computer equipment and storage medium | |
| CN108389124B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN112363937A (en) | Method and device for testing differential coverage rate, computer equipment and storage medium | |
| CN111723097A (en) | Application program interface configuration method and device, computer equipment and storage medium | |
| CN108776642B (en) | Test report generation method and device, computer equipment and storage medium | |
| CN114385485A (en) | Interface test method, device, equipment and storage medium | |
| CN108196975B (en) | Data verification method and device based on multiple checksums and storage medium | |
| CN111611002B (en) | Data upgrading method, device, computer equipment and storage medium | |
| CN115277678B (en) | File downloading method, device, computer equipment and storage medium | |
| CN110647336A (en) | Gray scale distribution method and device, computer equipment and storage medium | |
| CN112860702A (en) | Test execution work order verification method and device, computer equipment and storage medium | |
| CN111191982B (en) | Order grouping method and device, computer equipment and storage medium | |
| CN106326310B (en) | Resource encryption updating method for mobile phone client software | |
| CN110908599B (en) | Data writing method and system | |
| CN111538651A (en) | Interface testing method, device, server and storage medium | |
| CN112783866A (en) | Data reading method and device, computer equipment and storage medium | |
| CN111984527A (en) | Software performance testing method, device, equipment and medium | |
| CN110147384B (en) | Data search model establishment method, device, computer equipment and storage medium | |
| CN112579591A (en) | Data verification method and device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |