CN111290998A - Method, device and equipment for calibrating migration data and storage medium - Google Patents

Method, device and equipment for calibrating migration data and storage medium Download PDF

Info

Publication number
CN111290998A
CN111290998A CN202010091715.8A CN202010091715A CN111290998A CN 111290998 A CN111290998 A CN 111290998A CN 202010091715 A CN202010091715 A CN 202010091715A CN 111290998 A CN111290998 A CN 111290998A
Authority
CN
China
Prior art keywords
sub
file
migrated
verified
digest
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010091715.8A
Other languages
Chinese (zh)
Inventor
兰东平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN202010091715.8A priority Critical patent/CN111290998A/en
Priority to PCT/CN2020/093187 priority patent/WO2021159639A1/en
Publication of CN111290998A publication Critical patent/CN111290998A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/11File system administration, e.g. details of archiving or snapshots
    • G06F16/119Details of migration of file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1004Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's to protect a block of data words, e.g. CRC or checksum

Abstract

The invention provides a method, a device, equipment and a storage medium for calibrating migration data, namely, a file to be migrated before migration and a file to be verified after migration in a cloud are obtained in a source end; calculating sub-standard digest hashed values corresponding to the sub-files to be migrated, and calculating a total standard digest hashed value corresponding to the files to be migrated based on the sub-standard digest hashed values and the sizes of the sub-files to be migrated; calculating sub-to-be-verified digest hash values corresponding to the sub-files to be verified, and calculating a total to-be-verified digest hash value corresponding to the files to be verified based on the sub-to-be-verified digest hash values and the sizes of the sub-files to be verified; and checking the file to be checked, and if the total standard digest hashed value is the same as the total digest hashed value to be checked, judging that the file to be checked is successfully migrated. The invention reduces the comparison times of the files, improves the file migration efficiency and reduces the migration cost.

Description

Method, device and equipment for calibrating migration data and storage medium
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a computer-readable storage medium for calibrating migrated data.
Background
With the popularization of cloud computing, more and more users need to migrate data to the cloud for storage, particularly object storage. The amount of data migrated is large, reaching TB and even PB levels. In the process of uploading data to the object storage, errors such as missing migration, mismigration, data packet loss in network transmission and the like may occur. In order to ensure the completeness of data, data reconciliation is necessarily required. The traditional method for checking account of data usually compares the data before and after uploading one by one. Under different scenes, the comparison method is different. If the size of the file is compared one by one, and the MD5(Message-Digest Algorithm) is compared, once a file is found to be incorrect, the original file needs to be uploaded again, and the above data checking steps are repeated again until all the files are finished. Although the method of comparing files one by one can ensure the accuracy, the method of comparing files one by one can ensure the low efficiency of data proofreading and increase the cost of data proofreading because the data proofreading is carried out one by one and the data proofreading needs to transmit the file size and the MD5 through the network every time of data proofreading. Therefore, how to solve the technical problems of low data proofreading efficiency and high data proofreading cost in the prior art becomes a technical problem to be solved urgently at present.
Disclosure of Invention
The invention mainly aims to provide a method, a device and equipment for calibrating migrated data and a computer readable storage medium, aiming at solving the technical problems of low data calibration efficiency and higher data calibration cost in the prior art.
In order to achieve the above object, the present invention provides a method for checking migration data, including the following steps:
when a data migration instruction is detected, acquiring a file to be migrated before migration and a file to be verified after migration in a cloud end in a source end;
calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and calculating total standard digest hashed values corresponding to the files to be migrated based on the sub-standard digest hashed values and the sizes of the sub-files to be migrated;
calculating sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculating a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified;
and checking the file to be checked according to the total standard digest hash value and the total digest hash value to be checked, if the total standard digest hash value is the same as the total digest hash value to be checked, judging that the file to be checked is successfully migrated, and generating a corresponding file migration success reminding message.
Optionally, before the step of acquiring, in the source end, the file to be migrated before the migration and the file to be verified after the migration in the cloud when the data migration instruction is detected, the method further includes:
and sequentially arranging each subfile to be migrated in the source end according to a preset arrangement rule, and sequentially uploading each arranged subfile to be migrated to the cloud end.
Optionally, the step of calculating the sub-standard digest hash value corresponding to each sub-file to be migrated in the file to be migrated, and calculating the total standard digest hash value corresponding to the file to be migrated based on each sub-standard digest hash value and the size of each sub-file to be migrated specifically includes:
calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and setting the sizes of the sub-files to be migrated corresponding to the sub-files to be migrated and the sub-standard digest hashed values as sub-standard meta information corresponding to the sub-files to be migrated;
sequentially splicing sub-standard meta-information corresponding to a preset number of the sub-files to be migrated according to the arrangement sequence of the sub-files to be migrated to generate standard batch meta-information;
and calculating batch standard digest hashed values corresponding to the standard batch meta information, and calculating a total standard digest hashed value corresponding to the file to be migrated based on the batch standard digest hashed values.
Optionally, the step of calculating the sub-to-be-verified digest hash value corresponding to each sub-file to be verified in the file to be verified, and based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified, calculating the total to-be-verified digest hash value corresponding to the file to be verified specifically includes:
calculating sub-to-be-verified digest hashed values corresponding to each sub-file to be verified in the file to be verified, and setting the size of the sub-file to be verified corresponding to the sub-file to be verified and the sub-to-be-verified digest hashed values as sub-to-be-verified meta information corresponding to the sub-file to be verified;
sequentially splicing the sub-to-be-verified meta-information corresponding to the sub-files to be verified of the preset number according to the arrangement sequence of the sub-files to generate batch sub-to-be-verified meta-information;
calculating the batch to-be-verified digest hash values corresponding to the batch sub to-be-verified meta information, sequentially splicing the batch to-be-verified digest hash values for a preset number of times based on the batch to-be-verified digest hash values, and calculating a file to-be-verified digest hash value as a total to-be-verified digest hash value corresponding to the file to be verified.
Optionally, the checking the to-be-checked file according to the total standard digest hash value and the total to-be-checked digest hash value, and if the total standard digest hash value is the same as the total to-be-checked digest hash value, determining that the to-be-checked file is successfully migrated, and generating a corresponding file migration success reminding message specifically includes:
judging whether the total standard digest hash value is the same as the total to-be-verified digest hash value or not so as to perform file verification on the file to be verified after the file is migrated in the cloud;
if the total standard digest hash value is different from the total digest hash value to be verified, comparing the batch standard digest hash values with corresponding batch digest hash values to be verified so as to verify batch subfiles with abnormal migration;
and if the total standard digest hash value is the same as the total digest hash value to be verified, judging that the file to be verified is successfully migrated, and generating a corresponding file migration success reminding message.
Optionally, the sequentially arranging the subfiles to be migrated in the source end according to a preset arrangement rule, and sequentially uploading the arranged subfiles to be migrated to the cloud end specifically includes:
and sequentially arranging each subfile to be migrated in the source end according to the dictionary sequence of the file name, and sequentially uploading each arranged subfile to be migrated to the cloud.
In addition, to achieve the above object, the present invention further provides a verification apparatus for migration data, including:
the migration file determining module is used for acquiring a file to be migrated before migration and a file to be verified after migration in the cloud in the source end when a data migration instruction is detected;
the first MD5 calculation module is configured to calculate sub-standard digest hash values corresponding to the sub-files to be migrated in the files to be migrated, and calculate a total standard digest hash value corresponding to the files to be migrated based on each sub-standard digest hash value and the size of each sub-file to be migrated;
the second MD5 calculation module is configured to calculate sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculate a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified;
and the migration data checking module is used for checking the file to be checked according to the total standard digest hashed value and the total digest hashed value to be checked, judging that the file to be checked is successfully migrated if the total standard digest hashed value is the same as the total digest hashed value to be checked, and generating a corresponding file migration success reminding message.
Optionally, the verification apparatus for migration data further includes a migration file splicing module, where the migration file splicing module is configured to:
sequentially arranging each subfile to be migrated in the source end according to a preset arrangement rule, and sequentially uploading each arranged subfile to be migrated to the cloud end;
calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and setting the sizes of the sub-files to be migrated corresponding to the sub-files to be migrated and the sub-standard digest hashed values as sub-standard meta information corresponding to the sub-files to be migrated;
sequentially splicing sub-standard meta-information corresponding to a preset number of the sub-files to be migrated according to the arrangement sequence of the sub-files to be migrated to generate standard batch meta-information;
and calculating batch standard digest hashed values corresponding to the standard batch meta information, and calculating a total standard digest hashed value corresponding to the file to be migrated based on the batch standard digest hashed values.
In addition, in order to achieve the above object, the present invention further provides a device for collating migration data, where the device for collating migration data includes a processor, a memory, and a program for collating migration data stored on the memory and executable by the processor, where when the program for collating migration data is executed by the processor, the steps of the method for collating migration data as described above are implemented.
In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, where a collation program of migration data is stored, where when the collation program of migration data is executed by a processor, the steps of the collation method of migration data as described above are implemented.
The invention provides a verification method of migration data, which comprises the steps of acquiring a file to be migrated before migration and a file to be verified after migration in a cloud end in a source end when a data migration instruction is detected; calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and calculating total standard digest hashed values corresponding to the files to be migrated based on the sub-standard digest hashed values and the sizes of the sub-files to be migrated; calculating sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculating a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified; and checking the file to be checked according to the total standard digest hash value and the total digest hash value to be checked, if the total standard digest hash value is the same as the total digest hash value to be checked, judging that the file to be checked is successfully migrated, and generating a corresponding file migration success reminding message. Through the mode, according to the invention, the total standard digest hash value corresponding to the file to be migrated and the total digest hash value corresponding to the file to be verified are generated by respectively splicing and calculating through each sub-standard digest hash value of each sub-file to be migrated in the source end and each sub-file to-be-verified standard MD5 value of each sub-file to-be-verified sub-file in the file to be verified after migration in the cloud, and whether file migration is abnormal or not is determined by directly comparing the total standard digest hash value and the total digest hash value to be verified, so that the file comparison times are reduced, the file migration efficiency is improved, the migration cost is reduced, and the technical problems of low data verification efficiency and high data verification cost in the prior art are solved.
Drawings
Fig. 1 is a schematic hardware configuration diagram of a verification device for migration data according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a first embodiment of a migration data checking method according to the present invention;
FIG. 3 is a flowchart illustrating a second embodiment of a verification method for migrated data according to the present invention;
FIG. 4 is a flowchart illustrating a third embodiment of a migration data checking method according to the present invention;
FIG. 5 is a functional block diagram of a verification apparatus for migrated data according to a first embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The method for proofreading the migration data according to the embodiment of the present invention is mainly applied to a proofreading device for migration data, and the proofreading device for migration data may be a device having display and processing functions, such as a PC, a portable computer, a mobile terminal, and the like.
Referring to fig. 1, fig. 1 is a schematic diagram of a hardware structure of a verification device for migration data according to an embodiment of the present invention. In this embodiment of the present invention, the device for collating migration data may include a processor 1001 (e.g., a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used for realizing connection communication among the components; the user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface); the memory 1005 may be a high-speed RAM memory, or may be a non-volatile memory (e.g., a magnetic disk memory), and optionally, the memory 1005 may be a storage device independent of the processor 1001.
Those skilled in the art will appreciate that the hardware configuration shown in FIG. 1 does not constitute a limitation of a collating device migrating data, and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
With continued reference to fig. 1, a memory 1005 of fig. 1, which is one type of computer-readable storage medium, may include an operating system, a network communication module, and a collation program for migrating data.
In fig. 1, the network communication module is mainly used for connecting to a server and performing data communication with the server; the processor 1001 may call a collation program of the migration data stored in the memory 1005, and execute the collation method of the migration data according to the embodiment of the present invention.
The embodiment of the invention provides a method for checking migration data.
Referring to fig. 2, fig. 2 is a flowchart illustrating a first embodiment of a verification method for migrated data according to the present invention.
In this embodiment, the method for checking the migration data includes the following steps:
step S10, when a data migration instruction is detected, acquiring a file to be migrated before migration and a file to be verified after migration in the cloud from the source end;
the traditional method for checking account of data usually compares the data before and after uploading one by one. Under different scenes, the comparison method is different. If the size of the file is compared one by one, and the MD5(Message-Digest Algorithm) is compared, once a file is found to be incorrect, the original file needs to be uploaded again, and the above data checking steps are repeated again until all the files are finished. Although the method of comparing files one by one can ensure the accuracy, the method of comparing files one by one can ensure the low efficiency of data proofreading and increase the cost of data proofreading because the data proofreading is carried out one by one and the data proofreading needs to transmit the file size and the MD5 through the network every time of data proofreading. In order to solve the above problems, the present invention generates a total standard MD5 corresponding to the file to be migrated, that is, a total standard digest hash value, and a total to-be-verified MD5 corresponding to the file to be verified, that is, a total to-be-verified digest hash value, by respectively concatenating and calculating the sub-file standard MD5 values, that is, the sub-standard digest hash values, of each sub-file to be migrated in the source end and the sub-file to-be-verified standard MD5 values of each sub-file to be verified in the file to be verified after migration in the cloud, and determines whether the file migration is abnormal by directly comparing the total standard MD5, that is, the total standard digest hash value, and the total to-be-verified MD5, that is, the total to-verified digest hash value, thereby reducing the number of file comparison, improving the.
Before step S10, the method further includes:
and sequentially arranging each subfile to be migrated in the source end according to a preset arrangement rule, and sequentially uploading each arranged subfile to be migrated to the cloud end.
And sequentially arranging each subfile to be migrated in the source end according to the dictionary sequence of the file names, and sequentially uploading each arranged subfile to be migrated to the cloud. Specifically, MD5(Message-Digest Algorithm, MD5 Message Digest Algorithm) is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value (hash value) to ensure the integrity of the Message transmission. The files to be migrated are arranged in a plurality of orders, for example, if the files to be migrated are under the respective directories, that is, the storage paths of the files are different, the files are sorted according to the lexicographic order of the respective file names, for example, the file1 is ranked before the file 2. And executing a migration program of the files to be migrated according to the arranged file sequence, namely uploading the files to be migrated to the cloud end one by one from the source end according to the requirement for storage.
Step S20, calculating sub-standard digest hash values corresponding to each sub-standard digest hash value in the file to be migrated, and calculating a total standard digest hash value corresponding to the file to be migrated based on each sub-standard digest hash value and the size of each sub-standard digest hash value;
in this embodiment, in the uploading process, a subfile standard MD5, that is, a sub-standard digest hash value, corresponding to each subfile to be migrated in the file to be migrated is calculated in the source end, and a standard MD5 corresponding to each subfile to be migrated is stored. The standard MD5 corresponding to the sub-file to be migrated may be MD5 calculated only according to the size of the file to be migrated, or may be MD5 calculated according to the size of the file to be migrated, and then the standard MD5 and the MD5 to be verified are further calculated according to the size of the file to be migrated and the MD5 corresponding to the file to be migrated.
Step S30, calculating sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculating a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified;
in this embodiment, the files to be checked that are migrated to the cloud are calculated with reference to the calculation step in step S20, the sub-to-be-checked digest hash values corresponding to the sub-files to be checked in the files to be checked are calculated, and the MD5 to be checked corresponding to the sub-files to be checked in the sub-files to be checked is stored.
Step S40, the file to be verified is verified according to the total standard digest hash value and the total digest hash value to be verified, if the total standard digest hash value is the same as the total digest hash value to be verified, it is determined that the file to be verified is successfully migrated, and a corresponding file migration success reminding message is generated.
In this embodiment, by directly comparing the total standard MD5 of the source end and the cloud end, that is, the total standard digest hash value with the total to-be-verified MD5, that is, the total to-be-verified digest hash value, if the two values are identical, it is verified that all the migration operations are correct. Otherwise, further checking the subfiles.
The embodiment provides a method for calibrating migration data, which includes acquiring a file to be migrated before migration and a file to be calibrated after migration in a cloud in a source end when a data migration instruction is detected; calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and calculating total standard digest hashed values corresponding to the files to be migrated based on the sub-standard digest hashed values and the sizes of the sub-files to be migrated; calculating sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculating a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified; and checking the file to be checked according to the total standard digest hash value and the total digest hash value to be checked, if the total standard digest hash value is the same as the total digest hash value to be checked, judging that the file to be checked is successfully migrated, and generating a corresponding file migration success reminding message. Through the mode, according to the invention, the total standard digest hash value corresponding to the file to be migrated and the total digest hash value corresponding to the file to be verified are generated by respectively splicing and calculating through each sub-standard digest hash value of each sub-file to be migrated in the source end and each sub-file to-be-verified standard MD5 value of each sub-file to-be-verified sub-file in the file to be verified after migration in the cloud, and whether file migration is abnormal or not is determined by directly comparing the total standard digest hash value and the total digest hash value to be verified, so that the file comparison times are reduced, the file migration efficiency is improved, the migration cost is reduced, and the technical problems of low data verification efficiency and high data verification cost in the prior art are solved.
Referring to fig. 3, fig. 3 is a flowchart illustrating a second embodiment of the verification method for migrated data according to the present invention.
Based on the foregoing embodiment shown in fig. 2, in this embodiment, the step S20 specifically includes:
step S21, calculating sub-standard digest hashed values corresponding to each sub-file to be migrated in the file to be migrated, and setting the size of the sub-file to be migrated corresponding to the sub-file to be migrated and the sub-standard digest hashed values as sub-standard meta information corresponding to the sub-file to be migrated;
step S22, sequentially splicing the sub-standard meta-information corresponding to the sub-files to be migrated in a preset number according to the arrangement sequence of the sub-files to generate standard batch meta-information;
step S23, calculating a batch standard digest hash value corresponding to the standard batch meta information, and calculating a total standard digest hash value corresponding to the file to be migrated based on the batch standard digest hash value.
Wherein, the step S23 specifically includes:
and calculating batch standard digest hash values corresponding to the standard batch meta information, sequentially splicing the batch standard digest hash values for a preset number of times, and calculating a file standard MD5 as a total standard digest hash value corresponding to the file to be migrated.
In this embodiment, the processing procedure of the MD5 algorithm is as follows: and processing the file to be migrated, setting an initial value, circularly processing and splicing results. The MD5 algorithm is specifically described as follows:
firstly, processing a file to be migrated;
first, we calculate the length (bit) of the file to be migrated as the remainder of 512, and if not equal to 448, we need to fill the file to be migrated so that the length of the file to be migrated as the remainder of 512 is equal to 448. The method of padding is that the first bit is padded with 1 and the rest of the bits are padded with 0. After filling, the length of the file to be migrated is 512 × N + 448.
Then, the real length of the file to be migrated is recorded by the remaining position (512-. The length of the file to be migrated is 512 x (N + 1).
Secondly, setting an initial value;
the hash result of MD5 is 128 bits long and is divided into 4 groups of 32 bits each. These 4 sets of results are derived from 4 initial values A, B, C, D over time. In the official implementation of MD5, the initial values of A, B, C, D are as follows (16 scale):
A=0x01234567、B=0x89ABCDEF、C=0xFEDCBA98、D=0x76543210
the third step: circularly processing;
a, B, C, D are the four packets of hash values. Each cycle allows the old ABCD to generate a new ABCD. How many cycles in total? The length of the processed file to be migrated is determined.
Assuming that the length of the processed file to be migrated is M, the number of main cycles is M/512, and each main cycle includes 512/32 × 4 and 64 sub-cycles.
The fourth step: splicing results;
splicing four values A, B, C and D finally generated by circular processing together, and converting the spliced four values into character strings.
And taking the MD5 calculated by each file as the meta-information of the file, and splicing the meta-information of a preset number one by one according to a preset splicing unit in the sorting sequence of the files. For example, the meta information of each 1000 files is spliced into a total file (which is properly adjusted according to the specific number of the files to be migrated, minimum 10, maximum 1000), the total file, i.e. the additionally accumulated meta information, is further calculated into a total standard MD5, i.e. a total standard digest hash value, and a total to-be-verified MD5, i.e. a total to-be-verified digest hash value, i.e. MD5 — 1000. As described in detail below.
The first file1, MD5 value is MD5_ file 1;
a second file2, MD5 value MD5_ file 2;
a third file3, with an MD5 value of MD5_ file 3;
and so on, up to 1000 files.
And the splicing meta-information of MD5_1000 is: MD5_ file1MD5_ file2MD5_ file3 … MD5_ file 1000. Then, 1000 MD5_1000 files are gradually spliced and accumulated, and each 1000 MD5_1000 files are calculated to generate MD5_1000_ 1000. Through the above repeated steps, the source end and the cloud end both finally generate an MD5 value calculated through a limited number of splicing and accumulation, which is assumed to be MD5_1000_1000_1000 (corresponding to 100 ten thousand file migration scenarios), i.e., the total standard MD5, i.e., the total standard digest hash value, and the total MD5 to be verified, i.e., the total digest hash value to be verified.
Further, after the data migration of all the files to be migrated is completed, the data reconciliation operation is started. There are only two results, data migration is all correct (high probability), and the data migration process is wrong.
Further, the step S30 specifically includes:
calculating sub-to-be-verified digest hashed values corresponding to each sub-file to be verified in the file to be verified, and setting the size of the sub-file to be verified corresponding to the sub-file to be verified and the sub-to-be-verified digest hashed values as sub-to-be-verified meta information corresponding to the sub-file to be verified;
sequentially splicing the sub-to-be-verified meta-information corresponding to the sub-files to be verified of the preset number according to the arrangement sequence of the sub-files to generate batch sub-to-be-verified meta-information;
calculating the batch to-be-verified MD5 (batch to-be-verified digest hash value) corresponding to the batch sub-to-be-verified meta information, and calculating a file to-be-verified MD5 (file to-be-verified digest hash value) as a total to-be-verified MD5 (total to-be-verified digest hash value) corresponding to the file to be verified based on the batch to-be-verified MD5 (batch to-be-verified digest hash value) and sequentially splicing the batch to-be-verified MD5 (batch to-be-verified digest hash value) for a preset number of times.
In this embodiment, the total digest hash value to be verified, i.e., the total MD5 to be verified, corresponding to the file to be verified is calculated through the above steps.
Referring to fig. 4, fig. 4 is a flowchart illustrating a third embodiment of a verification method for migrated data according to the present invention.
Based on the foregoing embodiment shown in fig. 3, in this embodiment, the step S40 specifically includes:
step S41, determining whether the total standard digest hash value is the same as the total to-be-verified digest hash value, so as to perform file verification on the to-be-verified file migrated in the cloud;
step S42, if the total standard digest hash value is different from the total digest hash value to be verified, comparing the batch standard digest hash values with the corresponding batch digest hash values to be verified so as to verify batch subfiles with abnormal migration;
step S43, if the total standard digest hash value is the same as the total to-be-verified digest hash value, determining that the file to be verified is successfully migrated, and generating a corresponding file migration success reminding message.
In this embodiment, by directly comparing the file to be migrated at the source end with the file to be checked MD5_1000_1000 at the cloud end, if the two values are identical, it is verified that all the files are migrated correctly. Only one request for account checking occurs at this time. If the two values are not the same, the wrong file needs to be found quickly. The process of searching for the error file needs to be performed in reverse. That is, after comparing MD5_1000_1000_1000 differently, the next layer compares 1000 MD5_1000_1000 to find out the values of different sources and clouds. To improve efficiency, the 1000 MDs 5 — 1000 can be batch-checked and sent to a checking program for comparison. By circulating the above steps, files with different values of MD5, i.e. files with migration errors in the migration process, can be quickly found. And quickly performs retransmission.
Therefore, on the premise of ensuring the accuracy, the quick account checking can be realized; meanwhile, if the file is migrated wrongly, the file can be found out quickly and migrated; the network cost is very high, thus the network transmission can be greatly reduced, and the cost is reduced.
For example, when the number of files to be migrated is 100 ten thousand.
Conventionally, the source and cloud files MD5 need to be compared 100 ten thousand times.
The efficient method is used for the following conditions:
1. if the migration is completely correct, the comparison needs to be carried out for 1 time;
2. if the migration process has 1 file error, 1000 times of comparison are needed for MD 5-1000 + N times, and the error file is found out. (N1000-
3. Similarly, if 2 files fail in the migration process, the comparison needs to be performed 1000+2N times to find out the error file. (N1000-
4. Similarly, if the file fails more in the migration process, the comparison times are more. If about n files fail, then compare 1000+ nN times.
Generally, the probability of file migration errors is small, and the probability of occurrence of the first three types is higher. Namely, the efficiency is improved by nearly 1000 times. Moreover, most of the comparison is that on the migration server, only limited network transmission is needed. .
In addition, the embodiment of the invention also provides a device for calibrating the migration data.
Referring to fig. 5, fig. 5 is a functional module diagram of a proof reading apparatus for migration data according to a first embodiment of the present invention.
In this embodiment, the device for checking migration data includes:
the migration file determining module 10 is configured to, when a data migration instruction is detected, obtain a file to be migrated before migration and a file to be verified after migration in the cloud from the source end;
the first MD5 calculating module 20 is configured to calculate sub-standard digest hash values corresponding to the sub-files to be migrated in the files to be migrated, and calculate a total standard digest hash value corresponding to the files to be migrated based on the sub-standard digest hash values and the sizes of the sub-files to be migrated;
the second MD5 calculating module 30 is configured to calculate sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculate a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified;
and the migration data checking module 40 is configured to check the file to be checked according to the total standard digest hash value and the total digest hash value to be checked, and if the total standard digest hash value is the same as the total digest hash value to be checked, determine that the file to be checked is successfully migrated, and generate a corresponding file migration success reminding message.
Further, the device for checking migration data further includes:
and the file sequence module is used for sequentially arranging each sub-file to be migrated in the source end according to a preset arrangement rule and sequentially uploading each arranged sub-file to be migrated to the cloud end.
Further, the first MD5 calculation module 20 specifically includes:
the first standard calculating unit is used for calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and setting the sizes of the sub-files to be migrated corresponding to the sub-files to be migrated and the sub-standard digest hashed values as sub-standard meta information corresponding to the sub-files to be migrated;
the second standard calculation unit is used for sequentially splicing the sub-standard meta-information corresponding to the sub-files to be migrated in a preset number according to the arrangement sequence of the sub-files to generate standard batch meta-information;
and the third standard calculating unit is used for calculating batch standard digest hashed values corresponding to the standard batch meta information and calculating a total standard digest hashed value corresponding to the file to be migrated based on the batch standard digest hashed values.
Further, the third criterion calculation unit is further configured to:
and calculating batch standard digest hash values corresponding to the standard batch meta information, sequentially splicing the batch standard digest hash values for a preset number of times, and calculating a file standard MD5 as a total standard digest hash value corresponding to the file to be migrated.
Further, the second MD5 calculation module 30 specifically includes:
the first to-be-verified calculation unit is used for calculating sub to-be-verified digest hashed values corresponding to each to-be-verified sub-file in the to-be-verified file, and setting the size of the to-be-verified sub-file corresponding to the verification sub-file and the sub to-be-verified digest hashed values as sub to-be-verified meta information corresponding to the to-be-verified sub-file;
the second to-be-verified calculation unit is used for sequentially splicing the sub to-be-verified meta information corresponding to the preset number of the sub files to be verified according to the arrangement sequence of the sub files to generate batch sub to-be-verified meta information;
and the third to-be-verified calculating unit is used for calculating batch to-be-verified digest hashed values corresponding to the batch sub to-be-verified meta information, sequentially splicing the batch to-be-verified digest hashed values for a preset number of times based on the batch to-be-verified digest hashed values, and calculating a file to-be-verified digest hashed value as a total to-be-verified digest hashed value corresponding to the to-be-verified file.
Further, the migration data checking module 40 specifically includes:
the first MD5 proofreading unit is used for judging whether the total standard digest hash value is the same as the total to-be-verified digest hash value or not so as to perform file proofreading on the file to be verified after the file is migrated in the cloud;
the second MD5 checking unit is configured to compare the batch standard digest hash values with corresponding batch digest hash values to be checked if the total standard digest hash values are different from the total digest hash values to be checked, so as to verify batch subfiles with abnormal migration;
and the migration success reminding unit is used for judging that the file to be verified is successfully migrated if the total standard digest hashed value is the same as the total digest hashed value to be verified, and generating a corresponding file migration success reminding message.
Each module in the verification device for migration data corresponds to each step in the verification method for migration data, and the functions and implementation processes thereof are not described in detail herein.
In addition, the embodiment of the invention also provides a computer readable storage medium.
The computer readable storage medium of the present invention stores a verification program of migration data, wherein when the verification program of migration data is executed by a processor, the steps of the verification method of migration data as described above are implemented.
The method for implementing the verification procedure of the migration data when executed may refer to each embodiment of the verification method of the migration data of the present invention, and details thereof are not described herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (10)

1. A method for checking migrated data, the method comprising:
when a data migration instruction is detected, acquiring a file to be migrated before migration and a file to be verified after migration in a cloud end in a source end;
calculating standard sub-digest hashed values corresponding to each sub-file to be migrated in the file to be migrated, and calculating a standard total digest hashed value corresponding to the file to be migrated based on the standard sub-digest hashed values and the size of each sub-file to be migrated;
calculating sub-to-be-verified digest hash values corresponding to the sub-files to be verified in the files to be verified, and calculating total digest hash values to be verified corresponding to the files to be verified based on the sub-digest hash values to be verified and the sizes of the sub-files to be verified;
and checking the file to be checked according to the total standard digest hash value and the total digest hash value to be checked, if the total standard digest hash value is the same as the total digest hash value to be checked, judging that the file to be checked is successfully migrated, and generating a corresponding file migration success reminding message.
2. The method for checking migration data according to claim 1, wherein before the step of acquiring, in the source end, the file to be migrated before the migration and the file to be checked after the migration in the cloud end when the data migration instruction is detected, the method further comprises:
and sequentially arranging each subfile to be migrated in the source end according to a preset arrangement rule, and sequentially uploading each arranged subfile to be migrated to the cloud end.
3. The method for checking migration data according to claim 2, wherein the step of calculating the sub-standard digest hash value corresponding to each sub-file to be migrated in the file to be migrated, and based on each sub-standard digest hash value and the size of each sub-file to be migrated, calculating the total standard digest hash value corresponding to the file to be migrated specifically includes:
calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and setting the sizes of the sub-files to be migrated corresponding to the sub-files to be migrated and the sub-standard digest hashed values as sub-standard meta information corresponding to the sub-files to be migrated;
sequentially splicing sub-standard meta-information corresponding to a preset number of the sub-files to be migrated according to the arrangement sequence of the sub-files to be migrated to generate standard batch meta-information;
and calculating a batch standard digest hashed value corresponding to the standard batch meta information, and calculating a standard total digest hashed value corresponding to the file to be migrated based on the standard batch digest hashed value.
4. The method for checking migrated data according to claim 3, wherein the step of calculating the sub-to-be-checked digest hash value corresponding to each sub-file to be checked in the file to be checked, and based on each sub-to-be-checked digest hash value and the size of each sub-file to be checked, the step of calculating the total to-be-checked digest hash value corresponding to the file to be checked specifically includes:
calculating sub-to-be-verified digest hashed values corresponding to each sub-file to be verified in the file to be verified, and setting the size of the sub-file to be verified corresponding to the sub-file to be verified and the sub-to-be-verified digest hashed values as sub-to-be-verified meta information corresponding to the sub-file to be verified;
sequentially splicing the sub-to-be-verified meta-information corresponding to the sub-files to be verified of the preset number according to the arrangement sequence of the sub-files to generate batch sub-to-be-verified meta-information;
calculating the batch to-be-verified digest hash values corresponding to the batch sub to-be-verified meta information, sequentially splicing the batch to-be-verified digest hash values for a preset number of times based on the batch to-be-verified digest hash values, and calculating a file to-be-verified digest hash value as a total to-be-verified digest hash value corresponding to the file to be verified.
5. The method according to claim 4, wherein the step of checking the file to be checked according to the total standard digest hash value and the total digest hash value to be checked, and if the total standard digest hash value is the same as the total digest hash value to be checked, determining that the file to be checked is successfully migrated, and generating a corresponding file migration success reminding message specifically includes:
judging whether the total standard digest hash value is the same as the total to-be-verified digest hash value or not so as to perform file verification on the file to be verified after the file is migrated in the cloud;
if the total standard digest hash value is different from the total digest hash value to be verified, comparing the batch standard digest hash values with corresponding batch digest hash values to be verified so as to verify batch subfiles with abnormal migration;
and if the total standard digest hash value is the same as the total digest hash value to be verified, judging that the file to be verified is successfully migrated, and generating a corresponding file migration success reminding message.
6. The method according to any one of claims 1 to 5, wherein the step of sequentially arranging the subfiles to be migrated in the source end according to a preset arrangement rule, and sequentially uploading the arranged subfiles to be migrated to the cloud specifically includes:
and sequentially arranging each subfile to be migrated in the source end according to the dictionary sequence of the file name, and sequentially uploading each arranged subfile to be migrated to the cloud.
7. A device for collating migration data, characterized by comprising:
the migration file determining module is used for acquiring a file to be migrated before migration and a file to be verified after migration in the cloud in the source end when a data migration instruction is detected;
the first MD5 calculation module is configured to calculate sub-standard digest hash values corresponding to the sub-files to be migrated in the files to be migrated, and calculate a total standard digest hash value corresponding to the files to be migrated based on each sub-standard digest hash value and the size of each sub-file to be migrated;
the second MD5 calculation module is configured to calculate sub-to-be-verified digest hash values corresponding to each sub-file to be verified in the file to be verified, and calculate a total to-be-verified digest hash value corresponding to the file to be verified based on each sub-to-be-verified digest hash value and the size of each sub-file to be verified;
and the migration data checking module is used for checking the file to be checked according to the total standard digest hashed value and the total digest hashed value to be checked, judging that the file to be checked is successfully migrated if the total standard digest hashed value is the same as the total digest hashed value to be checked, and generating a corresponding file migration success reminding message.
8. The apparatus for collating migration data according to claim 7, wherein said apparatus for collating migration data further includes a migration file splicing module, said migration file splicing module is configured to:
sequentially arranging each subfile to be migrated in the source end according to a preset arrangement rule, and sequentially uploading each arranged subfile to be migrated to the cloud end;
calculating sub-standard digest hashed values corresponding to the sub-files to be migrated in the files to be migrated, and setting the sizes of the sub-files to be migrated corresponding to the sub-files to be migrated and the sub-standard digest hashed values as sub-standard meta information corresponding to the sub-files to be migrated;
sequentially splicing sub-standard meta-information corresponding to a preset number of the sub-files to be migrated according to the arrangement sequence of the sub-files to be migrated to generate standard batch meta-information;
and calculating batch standard digest hashed values corresponding to the standard batch meta information, and calculating a total standard digest hashed value corresponding to the file to be migrated based on the batch standard digest hashed values.
9. A collation apparatus for migrated data, characterized in that the collation apparatus for migrated data includes a processor, a memory, and a collation program for migrated data stored on the memory and executable by the processor, wherein the collation program for migrated data, when executed by the processor, implements the steps of the collation method for migrated data according to any one of claims 1 to 6.
10. A computer-readable storage medium, on which a collation program of migration data is stored, wherein the collation program of migration data, when executed by a processor, implements the steps of the collation method of migration data according to any one of claims 1 to 6.
CN202010091715.8A 2020-02-12 2020-02-12 Method, device and equipment for calibrating migration data and storage medium Pending CN111290998A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202010091715.8A CN111290998A (en) 2020-02-12 2020-02-12 Method, device and equipment for calibrating migration data and storage medium
PCT/CN2020/093187 WO2021159639A1 (en) 2020-02-12 2020-05-29 Checking method, apparatus, and device for data migration, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010091715.8A CN111290998A (en) 2020-02-12 2020-02-12 Method, device and equipment for calibrating migration data and storage medium

Publications (1)

Publication Number Publication Date
CN111290998A true CN111290998A (en) 2020-06-16

Family

ID=71018436

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010091715.8A Pending CN111290998A (en) 2020-02-12 2020-02-12 Method, device and equipment for calibrating migration data and storage medium

Country Status (2)

Country Link
CN (1) CN111290998A (en)
WO (1) WO2021159639A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112286910A (en) * 2020-11-23 2021-01-29 中国农业银行股份有限公司 Data verification method and device
CN112714155A (en) * 2020-12-14 2021-04-27 国电南瑞科技股份有限公司 Electric power operation data consistency verification method and device based on end cloud cooperative service
CN115426290A (en) * 2022-09-23 2022-12-02 中国农业银行股份有限公司 Data migration and verification method and device, computer equipment and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104731792A (en) * 2013-12-19 2015-06-24 中国银联股份有限公司 Method and system for verifying database consistency and method and system for positioning data difference
CN107037978A (en) * 2016-10-31 2017-08-11 福建亿榕信息技术有限公司 Data Migration bearing calibration and system
CN110413441A (en) * 2019-06-18 2019-11-05 平安科技(深圳)有限公司 Active and standby storage volume synchrodata method of calibration, device, equipment and storage medium
CN110457628A (en) * 2019-07-05 2019-11-15 平安国际智慧城市科技股份有限公司 Webpage edition correcting method, device, equipment and storage medium

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103646082B (en) * 2013-12-12 2017-04-05 北京奇安信科技有限公司 A kind of method and device of file verification
CN106484690A (en) * 2015-08-24 2017-03-08 阿里巴巴集团控股有限公司 A kind of verification method of Data Migration and device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104731792A (en) * 2013-12-19 2015-06-24 中国银联股份有限公司 Method and system for verifying database consistency and method and system for positioning data difference
CN107037978A (en) * 2016-10-31 2017-08-11 福建亿榕信息技术有限公司 Data Migration bearing calibration and system
CN110413441A (en) * 2019-06-18 2019-11-05 平安科技(深圳)有限公司 Active and standby storage volume synchrodata method of calibration, device, equipment and storage medium
CN110457628A (en) * 2019-07-05 2019-11-15 平安国际智慧城市科技股份有限公司 Webpage edition correcting method, device, equipment and storage medium

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112286910A (en) * 2020-11-23 2021-01-29 中国农业银行股份有限公司 Data verification method and device
CN112286910B (en) * 2020-11-23 2024-04-12 中国农业银行股份有限公司 Data verification method and device
CN112714155A (en) * 2020-12-14 2021-04-27 国电南瑞科技股份有限公司 Electric power operation data consistency verification method and device based on end cloud cooperative service
CN115426290A (en) * 2022-09-23 2022-12-02 中国农业银行股份有限公司 Data migration and verification method and device, computer equipment and storage medium

Also Published As

Publication number Publication date
WO2021159639A1 (en) 2021-08-19

Similar Documents

Publication Publication Date Title
CN111290998A (en) Method, device and equipment for calibrating migration data and storage medium
CN109660356B (en) Data uplink method, device, equipment and computer readable storage medium
WO2018177190A1 (en) Method and device for synchronizing blockchain data
WO2021036810A1 (en) Evidence verification method, system, apparatus and device, and readable storage medium
CN110457628B (en) Webpage version checking method, device, equipment and storage medium
CN103888254A (en) Network information verification method and apparatus
CN109635256B (en) Method and device for verifying data
CN111078672B (en) Data comparison method and device for database
CN113157651B (en) Method, system, equipment and medium for renaming resource files of android project in batches
CN104079623B (en) Multistage cloud storage synchronisation control means and system
CN108804315B (en) Test method and device applied to dynamic development, electronic equipment and storage medium
CN112131609A (en) Merkle tree-based electric energy quality data exchange format file integrity verification method and system
CN115795560A (en) Method, device, equipment and medium for checking integrity of file across systems
CN109597913B (en) Method, device, storage medium and electronic equipment for aligning document pictures
CN111339551A (en) Data verification method and related device and equipment
CN114185808A (en) Automatic testing method and device, electronic equipment and computer readable storage medium
CN111694502A (en) Block chain data storage method, device, equipment and storage medium
CN110806979B (en) Interface return value checking method, device, equipment and storage medium
CN111835871A (en) Method and device for transmitting data file and method and device for receiving data file
CN113609215A (en) Data processing method, device, equipment and computer readable storage medium
CN106326310B (en) Resource encryption updating method for mobile phone client software
CN112559546B (en) Database synchronization method, database synchronization device, computer equipment and readable storage medium
CN112632946A (en) Method, apparatus, computer device and storage medium for automatic table building
CN114463045A (en) Material file uploading method, device, equipment and storage medium
CN113343314A (en) Data verification method and device for data flashing

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination