CN116094770A - Data consistency verification method, device, system and computer equipment - Google Patents
Data consistency verification method, device, system and computer equipment Download PDFInfo
- Publication number
- CN116094770A CN116094770A CN202211660104.6A CN202211660104A CN116094770A CN 116094770 A CN116094770 A CN 116094770A CN 202211660104 A CN202211660104 A CN 202211660104A CN 116094770 A CN116094770 A CN 116094770A
- Authority
- CN
- China
- Prior art keywords
- data
- test data
- target test
- sequence
- offset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/12—Applying verification of the received information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Hardware Design (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a data consistency verification method, a data consistency verification device, a data consistency verification system, computer equipment and a storage medium. The method comprises the following steps: uploading target test data; generating expected data based on the random sequence seed and the offset in the process of downloading the target test data; the expected data is full-scale reference data for verifying consistency of the target test data; and reading target test data, comparing the read target test data with expected data, and positioning the inconsistent data of the target test data and the expected data. By adopting the method, the random sequence seeds and the offset can be recorded in the metadata of the target test data, the client can generate the expected data while downloading the target test data, and then read the target test data and compare the target test data with the expected data, so that the real-time positioning output of the inconsistent data content is realized.
Description
Technical Field
The present invention relates to the field of object storage data measurement, and in particular, to a data consistency check method, apparatus, system, computer device, storage medium, and computer program product.
Background
With the development of data storage technology, object storage (Object storage) technology, which is a technology of storing and managing data in an unstructured format (referred to as an Object), has emerged. Modern organizations need to create and analyze large amounts of unstructured data, such as photographs, videos, emails, web pages, sensor data, and audio files. The verification of the correctness of the data is performed in the stage of downloading the object, but the test tool for verifying the correctness of the data needs to record the written content when uploading the object, and the function of verifying the data is realized by comparing the downloaded content with the recorded uploaded content when downloading the object.
In the conventional technology, the MD5 value of test data is calculated as the last 32 characters of the test data, the MD5 value is calculated when an object is downloaded, then the calculated MD5 is compared with the last 32 characters of the object content, and if the calculated MD5 and the last 32 characters of the object content are inconsistent, the data correctness check fails. The second scheme is that files are created locally, then the files are uploaded to an object storage system, MD5 values are calculated according to object contents when the objects are downloaded, then MD5 values of the uploaded files are calculated, the two MD5 values are compared, and if the two MD5 values are different, data correctness checking fails.
However, under the condition that the disk space of the server is limited, the current object storage test data consistency check method is difficult to locate inconsistent data.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data consistency check method, apparatus, system, computer device, computer readable storage medium, and computer program product.
In a first aspect, the present application provides a data consistency check method. The method comprises the following steps:
uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
generating expected data based on the random sequence seed and the offset during downloading of the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
and reading the target test data, comparing the read target test data with the expected data 5, and positioning the inconsistent data of the target test data and the expected data.
In the data consistency verification method, the random sequence seeds and the offset are recorded in the metadata of the target test data, the client can generate expected data while downloading the target test data, and then the target test data is read and compared with the expected data, so that the content inconsistent with the data can be positioned and output in real time.
0 in one embodiment, before the uploading the target test data, the method further includes:
acquiring a random sequence seed and an offset;
generating a test data sequence according to the random sequence seed;
performing re-splicing treatment on the test data sequence according to the offset to obtain a spliced test data sequence;
and 5, constructing and obtaining initial test data according to the plurality of spliced test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
In the embodiment, the target test data is generated through the random sequence seeds and the offset, so that the content of the target test data is ensured to be different, and the performance of the test tool can be ensured; seed and offset random sequences
The amount is recorded in metadata of the initial test data, and it is possible to generate expected 0 data for comparison at the time of downloading.
In one embodiment, the re-splicing the test data sequence according to the offset to obtain a spliced test data sequence includes:
determining, for each of the plurality of offsets, target data of data bits characterized by the offset in the test data sequence;
5 dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data;
and re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the sequence of the first test data sequence segment after, so as to obtain a spliced test data sequence corresponding to the offset.
In this embodiment, by repeatedly generating the test data sequence according to the test data sequence, the spliced test data sequence is obtained, so that the content of the target test data can be ensured to be different, the test data can be ensured to be different, and the generation efficiency of the target test data can be improved.
In one embodiment, the comparing the read target test data with the expected data, and locating inconsistent content according to a data verification result includes:
respectively reading the target test data and the expected data with preset data size;
comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent content according to the data verification result.
In this embodiment, by comparing the read target test data with the expected data and simultaneously outputting the content and the position of the expected data and the actual target test data, inconsistent content can be located.
In one embodiment, after locating inconsistent content according to the data verification result, the method further includes:
and comparing the data contained in the data verification result with inconsistent contents, and outputting the contents in a preset form.
In this embodiment, the inconsistent content and the inconsistent position can be visually displayed by outputting the inconsistent content of the data verification result in a preset form.
In a second aspect, the present application further provides a data consistency checking apparatus. The device comprises:
the uploading module is used for uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
the generation module is used for generating expected data based on the random sequence seeds and the offset in the process of downloading the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
And the comparison module is used for reading the target test data, comparing the read target test data with the expected data and positioning the inconsistent data of the target test data and the expected data.
In the data consistency verification device, the random sequence seeds and the offset are recorded in the metadata of the target test data, the client can generate expected data while downloading the target test data, and then the target test data is read and compared with the expected data, so that the content inconsistent with the data can be positioned and output in real time.
In one embodiment, before uploading the module, the apparatus further includes:
the acquisition module is used for acquiring the random sequence seeds and the offset;
the second generation module is used for generating a test data sequence according to the random sequence seed;
the re-splicing module is used for carrying out re-splicing treatment on the test data sequences according to the offset to obtain spliced test data sequences;
and the recording module is used for constructing and obtaining initial test data according to the plurality of test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
In the embodiment, the target test data is generated through the random sequence seeds and the offset, so that the content of the target test data is ensured to be different, and the performance of the test tool can be ensured; recording the random sequence seed and offset in the metadata of the initial test data may enable generation of expected data for comparison at download.
In one embodiment, the re-splicing module is specifically configured to:
determining, for each of the plurality of offsets, target data of data bits characterized by the offset in the test data sequence;
dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data;
and re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the sequence of the first test data sequence segment after, so as to obtain a spliced test data sequence corresponding to the offset.
In this embodiment, by repeatedly generating the test data sequence according to the test data sequence, the spliced test data sequence is obtained, so that the content of the target test data can be ensured to be different, the test data can be ensured to be different, and the generation efficiency of the target test data can be improved.
In one embodiment, the comparison module is specifically configured to:
respectively reading the target test data and the expected data with preset data size;
comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent content according to the data verification result.
In this embodiment, by comparing the read target test data with the expected data and simultaneously outputting the content and the position of the expected data and the actual target test data, inconsistent content can be located.
In one embodiment, after the comparison module, the apparatus further comprises:
and the output module is used for comparing the data contained in the data verification result with inconsistent contents and outputting the contents in a preset form.
In this embodiment, the inconsistent content and the inconsistent position can be visually displayed by outputting the inconsistent content of the data verification result in a preset form.
In a third aspect, the present application further provides a data consistency check system, the system including:
the client is used for uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the random sequence seed and the offset included in metadata of the target test data; generating expected data based on the random sequence seeds and the offset in the process of downloading the target test data; the expected data is the complete target test data; reading the target test data, comparing the read target test data with the expected data, and positioning inconsistent contents according to a data verification result;
And the server is used for receiving and storing the target test data.
In one embodiment, the server is specifically configured to determine that the target test data is successfully uploaded when the random sequence seed and the offset record are successful in the process of storing the target test data;
and when the random sequence seed and the offset are recorded successfully, determining that the target test data is failed to upload.
In this embodiment, whether the target test data is uploaded successfully is determined by recording the metadata of the target test data, and the client is not required to determine whether the target test data is uploaded successfully, but is implemented by the object storage system.
In a fourth aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
generating expected data based on the random sequence seed and the offset during downloading of the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
And reading the target test data, comparing the read target test data with the expected data, and positioning the inconsistent data of the target test data and the expected data.
In the computer equipment, the random sequence seeds and the offset are recorded in the metadata of the target test data, the client can generate expected data while downloading the target test data, and then the target test data is read and compared with the expected data, so that the real-time positioning output of inconsistent data content can be realized.
In one embodiment, before the uploading the target test data, before the method, the method further includes:
acquiring a random sequence seed and an offset;
generating a test data sequence according to the random sequence seed;
performing re-splicing treatment on the test data sequence according to the offset to obtain a spliced test data sequence;
and constructing and obtaining initial test data according to the plurality of test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
In the embodiment, the target test data is generated through the random sequence seeds and the offset, so that the content of the target test data is ensured to be different, and the performance of the test tool can be ensured; recording the random sequence seed and offset in the metadata of the initial test data may enable generation of expected data for comparison at download.
In one embodiment, the re-splicing the test data sequence according to the offset to obtain a spliced test data sequence includes:
determining, for each of the plurality of offsets, target data of data bits characterized by the offset in the test data sequence;
dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data;
and re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the sequence of the first test data sequence segment after, so as to obtain a spliced test data sequence corresponding to the offset.
In this embodiment, by repeatedly generating the test data sequence according to the test data sequence, the spliced test data sequence is obtained, so that the content of the target test data can be ensured to be different, the test data can be ensured to be different, and the generation efficiency of the target test data can be improved.
In one embodiment, the comparing the read target test data with the expected data, and locating inconsistent content according to a data verification result includes:
respectively reading the target test data and the expected data with preset data size;
comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent content according to the data verification result.
In this embodiment, by comparing the read target test data with the expected data and simultaneously outputting the content and the position of the expected data and the actual target test data, inconsistent content can be located.
In one embodiment, after locating inconsistent content according to the data verification result, the method further includes:
and comparing the data contained in the data verification result with inconsistent contents, and outputting the contents in a preset form.
In this embodiment, the inconsistent content and the inconsistent position can be visually displayed by outputting the inconsistent content of the data verification result in a preset form.
In a fifth aspect, the present application also provides a computer-readable storage medium. The computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
generating expected data based on the random sequence seed and the offset during downloading of the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
and reading the target test data, comparing the read target test data with the expected data, and positioning the inconsistent data of the target test data and the expected data.
In the computer readable storage medium, the client can generate expected data while downloading the target test data by recording the random sequence seed and the offset in the metadata of the target test data, and then read the target test data and compare the target test data with the expected data, so that real-time positioning output of inconsistent data can be realized.
In one embodiment, before the uploading the target test data, before the method, the method further includes:
acquiring a random sequence seed and an offset;
generating a test data sequence according to the random sequence seed;
performing re-splicing treatment on the test data sequence according to the offset to obtain a spliced test data sequence;
and constructing and obtaining initial test data according to the plurality of test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
In the embodiment, the target test data is generated through the random sequence seeds and the offset, so that the content of the target test data is ensured to be different, and the performance of the test tool can be ensured; recording the random sequence seed and offset in the metadata of the initial test data may enable generation of expected data for comparison at download.
In one embodiment, the re-splicing the test data sequence according to the offset to obtain a spliced test data sequence includes:
determining, for each of the plurality of offsets, target data of data bits characterized by the offset in the test data sequence;
Dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data;
and re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the sequence of the first test data sequence segment after, so as to obtain a spliced test data sequence corresponding to the offset.
In this embodiment, by repeatedly generating the test data sequence according to the test data sequence, the spliced test data sequence is obtained, so that the content of the target test data can be ensured to be different, the test data can be ensured to be different, and the generation efficiency of the target test data can be improved.
In one embodiment, the comparing the read target test data with the expected data, and locating inconsistent content according to a data verification result includes:
respectively reading the target test data and the expected data with preset data size;
comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent content according to the data verification result.
In this embodiment, by comparing the read target test data with the expected data and simultaneously outputting the content and the position of the expected data and the actual target test data, inconsistent content can be located.
In one embodiment, after locating inconsistent content according to the data verification result, the method further includes:
and comparing the data contained in the data verification result with inconsistent contents, and outputting the contents in a preset form.
In this embodiment, the inconsistent content and the inconsistent position can be visually displayed by outputting the inconsistent content of the data verification result in a preset form.
The method, the device, the computer equipment, the storage medium and the computer program product for checking consistency of the object storage test data can realize data check when downloading the object by recording seed (random sequence seed) for generating the test data into metadata of the object, meanwhile, the test data is calculated and generated in a memory, the disk space of a server is not occupied, and the consistency check can be carried out on the object storage test data under the condition of limited disk space of the server and the position of inconsistent data is displayed.
Drawings
FIG. 1 is an application environment diagram of a data consistency check method in one embodiment;
FIG. 2 is a flow chart of a method of data consistency check in one embodiment;
FIG. 3 is a flow chart of a target test data generation step in one embodiment;
FIG. 4 is a flowchart illustrating a step of performing a re-splicing process on a test data sequence to obtain the test data sequence in one embodiment;
FIG. 5 is a flow chart of the content steps for locating data inconsistencies between target test data and expected data in one embodiment;
FIG. 6 is a flow diagram of an example of data consistency check in one embodiment;
FIG. 7 is a block diagram of a data consistency check device in one embodiment;
fig. 8 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The data consistency verification method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the client 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The terminal 102 may be, but is not limited to, various personal computers, notebook computers, smart phones, and tablet computers. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.
In one embodiment, as shown in fig. 2, a data consistency checking method is provided, and the method is applied to the client in fig. 1 for illustration, and includes the following steps:
and step 202, uploading target test data.
Wherein the target test data is generated from a random sequence seed and an offset. The metadata of the target test data contains a random sequence seed and an offset.
In this embodiment, when a user needs to test the object storage system, a data consistency test is typically implemented in a pressure test tool. And the data consistency check firstly records the content of the uploaded test data, and then compares whether the content of the uploaded test data is consistent with the content of the downloaded test data after downloading the test data. In the scheme, the client firstly acquires target test data, then selects a preset random algorithm, and generates the target test data through the random algorithm, a random sequence seed and an offset. Then, the client records the random sequence seed and the offset for generating the target test data in the metadata of the target test data, and uploads the target test data to the server.
In step 204, expected data is generated based on the random sequence seed and the offset during the process of downloading the target test data.
Wherein the expected data is a full-scale reference data for verifying consistency of the target test data.
In this embodiment, the client downloads the target test data, and in the downloading process, firstly reads the random sequence seed and the offset from the metadata of the target test data, where the metadata includes the tag, the type, the size and other attributes of the target test data, and when the random sequence seed and the offset are read, the client generates the expected data according to the random sequence seed and the offset.
And 206, reading target test data, comparing the read target test data with expected data, and positioning the inconsistent data of the target test data and the expected data.
In this embodiment, the client reads the target test data, and simultaneously reads the expected data corresponding to the target test data, compares the read target test data with the expected data as the actual data, determines that the data comparison result is inconsistent under the condition that the data of the target test data is inconsistent with the data of the expected data, and determines the data position where the data comparison result is inconsistent, namely the content where the positioning data is inconsistent.
In the data consistency verification method, the random sequence seeds and the offset are recorded in the metadata of the target test data, the client can generate expected data while downloading the target test data, and then the target test data is read and compared with the expected data, so that the content inconsistent with the data can be positioned and output in real time.
In one embodiment, as shown in FIG. 3, prior to uploading the targeted test data at step 202, it includes:
In this embodiment, the client obtains a random sequence seed and an offset, where the random sequence seed is an initial value used by a random algorithm to generate a random number, and the random sequence seed may be a current system time, and for a pseudo-random number generator, the same random number sequence may be obtained according to the same random sequence seed; the offset is the number of characters that move backwards from the specified position, different random sequences can be generated using the offset, and the client can generate a random number as the offset. The same content can be generated at the time of downloading by this random sequence seed and offset.
In this embodiment, the client first obtains a random sequence seed, then iterates with the random sequence seed as an initial value, and generates a test data sequence by using a random algorithm, where the random algorithm may be a square-median method or a linear congruence method, and the embodiment is not limited to this random algorithm.
And 306, performing re-splicing processing on the test data sequence according to the offset to obtain a spliced test data sequence.
In this embodiment, the client divides the test data sequence according to the offset, and re-concatenates the divided test data sequence to obtain a test data sequence. And then dividing and splicing the test data sequence repeatedly based on different offsets to obtain a spliced test data sequence.
And 308, constructing and obtaining initial test data according to the plurality of spliced test data sequences, and recording random sequence seeds and offsets in metadata of the initial test data to obtain target test data.
In this embodiment, the client performs concatenation on a plurality of test data sequences to obtain initial test data, the initial test data is obtained by repeatedly generating the test data sequences, and a random sequence seed and an offset are recorded in metadata of the initial test data to obtain target test data.
Optionally, the splicing sequence of the plurality of data sequences at this time is the sequence of the offset of the metadata record of the target test data, so as to obtain the target test data.
In the embodiment, the target test data is generated through the random sequence seeds and the offset, so that the content of the target test data is ensured to be different, and the performance of the test tool can be ensured; recording the random sequence seed and offset in the metadata of the initial test data may enable generation of expected data for comparison at download.
In one embodiment, as shown in fig. 4, step 306 performs a re-splicing process on the test data sequence according to the offset, to obtain a spliced test data sequence, which includes:
in step 402, for each of a plurality of offsets, target data for a data bit characterized by the offset is determined in a test data sequence.
Wherein the offset characterizes a data bit of the target data.
In this embodiment, the client obtains a plurality of offsets, and for each offset, determines a data bit of the target data in the test data sequence, determines the data bit as a position of the offset, for example, the test data sequence is a random sequence of 256k, the offset is a random number n, determines a start bit of the target data as an nth bit of the next 256k random sequence, and then determines an nth bit character as the target data.
And step 404, dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data.
The first test data sequence segment is data taking target data as a splitting position.
In this embodiment, the client takes the position of the target data as the position for splitting the test data sequence, then takes the data with the target data as the split position as the first test data sequence segment, and takes the rest of the test data sequence as the second test data sequence segment. For example, the test data sequence is a 256k random sequence, the offset is a random number n, the character of the nth bit is used as the target data, the character from the beginning of the nth bit to the end of the random sequence is used as the second test data sequence segment, and the previous character from the beginning of the random sequence to the nth bit is used as the first test data sequence segment.
And step 406, re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the first test data sequence segment after to obtain a spliced test data sequence.
In this embodiment, the client performs re-splicing on the second test data sequence segment and the first test data sequence segment according to the sequence that the second test data sequence segment is before and the first test data sequence segment is after, so as to obtain a spliced test data sequence. For example, the test data sequence is a 256k random sequence, the offset is a random number n, the nth bit of target data is used as the start of the second test data sequence segment, and the 1 st to n th bits of the random sequence are the first test data sequence segment. Then, the client splices the first test data sequence segments according to the sequence of the first test data sequence segments before the second test data sequence segments, and the spliced test data sequence is obtained. Similarly, if there are multiple offsets, the test data sequences are re-spliced in the same manner to obtain multiple test data sequences.
In this embodiment, by repeatedly generating the test data sequence according to the test data sequence, the spliced test data sequence is obtained, so that the content of the target test data can be ensured to be different, the test data can be ensured to be different, and the generation efficiency of the target test data can be improved.
In one embodiment, as shown in fig. 5, step 206 reads the target test data, compares the read target test data with expected data, locates the content of the target test data inconsistent with the data of the expected data, and includes:
In this embodiment, the data amount at the time of reading the data is preset in the client, and then, at the time of performing the data consistency check, the client reads the target data and the expected data according to the data amount.
And step 504, comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent contents according to the data verification result.
In this embodiment, the client compares expected data with a preset data size with actual read target test data with the preset data size, and if the content of the expected data is inconsistent with that of the target test data, determines the position of the target test data according to the random sequence seed and the offset, and outputs the content and the position of the expected data and the actual target test data; if the expected data is consistent with the content of the target test data, the client returns successful verification of the data consistency.
In this embodiment, by comparing the read target test data with the expected data and simultaneously outputting the content and the position of the expected data and the actual target test data, inconsistent content can be located.
In one embodiment, step 206 reads the target test data, and compares the read target test data with expected data, and after locating the content of the target test data inconsistent with the data of the expected data, the method further includes:
and outputting the content with inconsistent data verification results in a preset form.
In this embodiment, the client outputs the content with inconsistent data verification results in a preset form. The output result comprises each item of read target test data and expected data corresponding to the target test data, and also comprises position information of the target test data.
Alternatively, the preset form may be hexadecimal, or may be output in another form, which is not limited in the embodiment of the present application.
In this embodiment, the inconsistent content and the inconsistent position can be visually displayed by outputting the inconsistent content of the data verification result in a preset form.
The embodiment of the application also provides an example of a data consistency verification method, as shown in fig. 6, specifically including the following steps:
Step 603, recording the random sequence seed and the offset in metadata of the target test data.
In step 604, for each of the plurality of offsets, target data for the data bits characterized by the offset is determined in the test data sequence.
And 607, constructing and obtaining initial test data according to the plurality of test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
In step 608, expected data is generated based on the random sequence seed and the offset during the process of downloading the target test data.
Step 610, comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and locating inconsistent content according to the data verification result.
In step 611, the data included in the data verification result is compared with inconsistent content, and output in a preset form.
In one embodiment, a data consistency check system, the system comprising:
the client is used for uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; a random sequence seed and an offset included in metadata of the target test data; generating expected data based on the random sequence seeds and the offset in the process of downloading target test data; the expected data is complete target test data; reading target test data, comparing the read target test data with expected data, and positioning inconsistent contents according to a data verification result;
And the server is used for receiving and storing the target test data.
In one embodiment, the server is specifically configured to determine that the target test data is successfully uploaded when the random sequence seed and the offset record are successful in the process of storing the target test data;
and when the random sequence seed and the offset are recorded successfully, determining that the target test data is failed to upload.
In this embodiment, the server determines that the uploading of the target test data is successful under the condition that the random sequence seed and the offset are recorded successfully; under the condition that the random sequence seed and the offset record fail, the target test data is determined to be failed to upload, and the target test data can be downloaded again by the client for verification.
In this embodiment, whether the target test data is uploaded successfully is determined by recording the metadata of the target test data, and the client is not required to determine whether the target test data is uploaded successfully, but is implemented by the object storage system.
It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides a data consistency checking device for realizing the data consistency checking method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation in the embodiments of the data consistency checking device or devices provided below may refer to the limitation of the data consistency checking method hereinabove, and will not be repeated herein.
In one embodiment, as shown in fig. 7, there is provided a data consistency checking apparatus, including: an uploading module 701, a generating module 702 and a comparing module 703, wherein:
an uploading module 701, configured to upload target test data, where the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
a generating module 702, configured to generate expected data based on the random sequence seed and the offset during a process of downloading the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
And the comparison module 703 is configured to read the target test data, compare the read target test data with the expected data, and locate the content of the inconsistency between the target test data and the expected data.
In one embodiment, before uploading the module, the apparatus further includes:
the acquisition module is used for acquiring the random sequence seeds and the offset;
the second generation module is used for generating a test data sequence according to the random sequence seed;
the re-splicing module is used for carrying out re-splicing treatment on the test data sequences according to the offset to obtain spliced test data sequences;
and the recording module is used for constructing and obtaining initial test data according to the plurality of test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
In one embodiment, the re-splicing module is specifically configured to:
determining, for each of the plurality of offsets, target data of data bits characterized by the offset in the test data sequence;
dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data;
And re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the sequence of the first test data sequence segment after, so as to obtain a spliced test data sequence corresponding to the offset.
In one embodiment, the comparison module is specifically configured to:
respectively reading the target test data and the expected data with preset data size;
comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent content according to the data verification result.
In one embodiment, after the comparison module, the apparatus further comprises:
and the output module is used for comparing the data contained in the data verification result with inconsistent contents and outputting the contents in a preset form.
The modules in the data consistency checking device can be realized in whole or in part by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 8. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data consistency check method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.
It will be appreciated by those skilled in the art that the structure shown in fig. 8 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor implementing the steps of the embodiments of the data consistency check method described above when executing the computer program.
In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor performs the steps of the embodiments of the data consistency check method described above.
In one embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the steps of the embodiments of the data consistency check method described above.
It should be noted that, user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (10)
1. A method for verifying data consistency, the method comprising:
uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
generating expected data based on the random sequence seed and the offset during downloading of the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
And reading the target test data, comparing the read target test data with the expected data, and positioning the inconsistent data of the target test data and the expected data.
2. The method of claim 1, wherein prior to uploading the targeted test data, the method further comprises:
acquiring a random sequence seed and an offset;
generating a test data sequence according to the random sequence seed;
performing re-splicing treatment on the test data sequence according to the offset to obtain a spliced test data sequence;
and constructing and obtaining initial test data according to the plurality of spliced test data sequences, and recording the random sequence seeds and the offset in metadata of the initial test data to obtain target test data.
3. The method according to claim 2, wherein the re-splicing the test data sequence according to the offset to obtain a spliced test data sequence includes:
determining, for each of the plurality of offsets, target data of data bits characterized by the offset in the test data sequence;
Dividing the test data sequence by taking the position of the target data as a splitting position to obtain a first test data sequence segment and a second test data sequence segment containing the target data;
and re-splicing the second test data sequence segment and the first test data sequence segment according to the sequence of the second test data sequence segment before and the sequence of the first test data sequence segment after, so as to obtain a spliced test data sequence corresponding to the offset.
4. The method of claim 1, wherein the reading the target test data and comparing the read target test data with the expected data, locating the content of the target test data inconsistent with the expected data, comprises:
respectively reading the target test data and the expected data with preset data size;
comparing the expected data with the preset data size with the target test data with the preset data size to obtain a data verification result, and positioning inconsistent content according to the data verification result.
5. The method of claim 1, wherein after locating inconsistent content according to the data verification result, further comprising:
And comparing the data contained in the data verification result with inconsistent contents, and outputting the contents in a preset form.
6. A data consistency check device, the device comprising:
the uploading module is used for uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the metadata of the target test data comprises the random sequence seeds and the offset;
the generation module is used for generating expected data based on the random sequence seeds and the offset in the process of downloading the target test data; the expected data is full-scale reference data for verifying consistency of the target test data;
and the comparison module is used for reading the target test data, comparing the read target test data with the expected data and positioning the inconsistent data of the target test data and the expected data.
7. A data consistency verification system, the system comprising:
the client is used for uploading target test data, wherein the target test data is generated by a random sequence seed and an offset; the random sequence seed and the offset included in metadata of the target test data; generating expected data based on the random sequence seeds and the offset in the process of downloading the target test data; the expected data is the complete target test data; reading the target test data, comparing the read target test data with the expected data, and positioning inconsistent contents according to a data verification result;
And the server is used for receiving and storing the target test data.
8. The system of claim 7, the server further configured to determine, during the storing of the target test data, that the target test data was successfully uploaded when the random sequence seed and offset record were successful;
and when the random sequence seed and the offset record fail, determining that the target test data is failed to upload.
9. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any of claims 1 to 6 when the computer program is executed.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211660104.6A CN116094770A (en) | 2022-12-23 | 2022-12-23 | Data consistency verification method, device, system and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211660104.6A CN116094770A (en) | 2022-12-23 | 2022-12-23 | Data consistency verification method, device, system and computer equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116094770A true CN116094770A (en) | 2023-05-09 |
Family
ID=86203623
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211660104.6A Pending CN116094770A (en) | 2022-12-23 | 2022-12-23 | Data consistency verification method, device, system and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116094770A (en) |
-
2022
- 2022-12-23 CN CN202211660104.6A patent/CN116094770A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110597552B (en) | Configuration method, device, equipment and storage medium of project continuous integrated pipeline | |
| CN113901395B (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN106201857A (en) | The choosing method of test case and device | |
| US20200387649A1 (en) | Systems and methods for simulating real-world io workloads in a parallel and distributed storage system | |
| CN116094770A (en) | Data consistency verification method, device, system and computer equipment | |
| CN116401298A (en) | Data processing method, device, computer equipment and storage medium | |
| CN114185566A (en) | Containerized deployment method, apparatus, computer device and storage medium | |
| CN112420168B (en) | Method, device, equipment and storage medium for writing data into database | |
| CN117056228A (en) | Page loading test method and device, computer equipment and storage medium | |
| CN116484820A (en) | Form configuration method and device and computer equipment | |
| CN115438605A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN117033172A (en) | Test data processing method, apparatus, device, storage medium and program product | |
| CN114780404A (en) | Performance test method, apparatus, storage medium and program product for archiving system | |
| CN116401323A (en) | Index data processing method and device and computer equipment | |
| CN117459519A (en) | Traceable file processing method, traceable file processing device, computer equipment and storage medium | |
| CN114328405A (en) | File processing method, file processing device, computer equipment, storage medium and program product | |
| CN115687074A (en) | Business system testing method and device, computer equipment and storage medium | |
| CN116880852A (en) | Code data checking method and device based on machine learning and computer equipment | |
| CN115437935A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN115270766A (en) | Data quality verification method for long text extraction result | |
| CN115344544A (en) | Automatic driving software synchronization method and device, storage medium and computer equipment | |
| CN117785808A (en) | File processing method, device, computer equipment and storage medium | |
| CN117742586A (en) | Data storage method, device, computer equipment and storage medium | |
| CN116861120A (en) | Webpage resource loading method and device, computer equipment and storage medium | |
| CN117435651A (en) | Test data processing method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |