CN110442474B - Data consistency verification method, system and data verification terminal - Google Patents
Data consistency verification method, system and data verification terminal Download PDFInfo
- Publication number
- CN110442474B CN110442474B CN201810410346.7A CN201810410346A CN110442474B CN 110442474 B CN110442474 B CN 110442474B CN 201810410346 A CN201810410346 A CN 201810410346A CN 110442474 B CN110442474 B CN 110442474B
- Authority
- CN
- China
- Prior art keywords
- data
- monotonicity
- verification
- verified
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012795 verification Methods 0.000 title claims abstract description 93
- 238000013524 data verification Methods 0.000 title claims abstract description 62
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000004590 computer program Methods 0.000 claims description 3
- 230000001174 ascending effect Effects 0.000 claims 4
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 238000011084 recovery Methods 0.000 description 5
- 230000003247 decreasing effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
- G06F11/2082—Data synchronisation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data consistency verification method, a system, a data verification terminal and a computer readable storage medium, wherein the method comprises the steps of obtaining a data group of a linked list to be verified of a data writing terminal; sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not; if the version numbers are incorrect, version numbers corresponding to the follow-up preset first number of IO data of the current IO data are sequentially acquired to form a monotonicity judging group; if the monotonicity judging group is in a first monotonicity state, the current IO data passes verification; if the second monotonic state is passed, the verification of the linked list to be verified is finished; if the third monotonic state exists, IO data is lost, and error reporting operation is executed; the method can judge whether the data of the current data group of the data writing end and the data verification end are consistent or not by judging the monotonicity of the monotonicity judging group formed by the IO data of the chain table to be verified, and achieves consistency verification under the condition that the data writing end and the data verification end are out of step.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data consistency verification method, a system, a data verification terminal, and a computer readable storage medium.
Background
The consistency check is an important means for ensuring the reliability of data, and at present, the writing end and the verification end are deployed simultaneously in the existing consistency check scheme, so that the synchronization of the writing end and the verification end is easy to realize, and the verification is realized on the premise that a data linked list is complete. The data link list must be complete, which is the precondition of the normal operation of the existing consistency verification scheme, that is, the synchronization between the writing end and the verification end must be ensured. However, in many application scenarios, synchronization between the writing end and the verification end is difficult to ensure. For example, in a disaster recovery system, a production end and a disaster recovery end are deployed in different devices across regions, and the production end and the disaster recovery end are connected through a high-delay network. The writing end and the verification end lack of a real-time communication mechanism, so that the writing end and the verification end are asynchronous, the writing end and the verification end are fast, and the writing end and the verification end are slow, so that a data link list is partially covered and is incomplete; the former is slow and the latter is fast, the writing end cannot catch up with the verifying end after the bottom disc is reached, and invalid data can be read. This results in the existing consistency verification scheme not working effectively.
Therefore, how to implement data consistency verification under the condition that the writing end and the verification end are not synchronous is a technical problem that needs to be solved by those skilled in the art.
Disclosure of Invention
The invention aims to provide a data consistency verification method, a system, a data verification terminal and a computer readable storage medium, which can realize consistency verification under the condition that a data writing terminal and a data verification terminal are out of step.
In order to solve the technical problems, the invention provides a data consistency verification method, which comprises the following steps:
acquiring a data group of a linked list to be verified of a data writing end;
sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not;
if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the follow-up preset first number of IO data of the current IO data to form a monotonicity judging group;
if the monotonicity judging group is in a first monotonicity state, the data writing end is slow, the data verifying end is fast, and the current IO data is verified;
if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished;
and if the monotonicity judging group is in a third monotonicity state, IO data is lost, and error reporting operation is executed.
Optionally, determining whether the version number of the current IO data is correct includes:
and judging whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
Optionally, sequentially acquiring version numbers corresponding to a preset first number of IO data subsequent to the current IO data to form a monotonicity determination group, including:
sequentially acquiring version numbers corresponding to the follow-up preset first number of IO data of the current IO data, and determining weight values corresponding to the version numbers;
forming monotonicity judgment groups by the weight values according to the acquisition sequence; and the weight value corresponding to the version number from high to low is the smallest, and the weight value corresponding to the version number is null.
Optionally, after the current IO data verification passes, the method further includes:
judging whether a monotonicity judging group is formed by version numbers of IO data corresponding to a second number of data groups preset in the follow-up linked list to be verified or not;
if yes, the data verification end overtime, and the error reporting operation is executed.
The invention also provides a data verification terminal, comprising:
the acquisition module is used for acquiring the data group of the linked list to be verified of the data writing end;
the version number verification module is used for sequentially acquiring the IO data of the data group and judging whether the version number of the current IO data is correct or not;
the monotonicity judging group acquisition module is used for sequentially acquiring version numbers corresponding to the follow-up preset first number of IO data of the current IO data to form a monotonicity judging group if the version numbers of the current IO data are incorrect;
the consistency verification module is used for enabling the data writing end to be slow and the data verification end to be fast if the monotonicity judging group is in a first monotonicity state, and enabling the current IO data to pass verification; if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished; and if the monotonicity judging group is in a third monotonicity state, IO data is lost, and error reporting operation is executed.
Optionally, the version number verification module specifically acquires the IO data of the data set in sequence, and judges whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
Optionally, the monotonicity determination group acquisition module includes:
the weight value acquisition unit is used for sequentially acquiring version numbers corresponding to the follow-up preset first number of IO data of the current IO data and determining weight values corresponding to the version numbers;
the monotonicity judgment group acquisition unit is used for forming the weight values into monotonicity judgment groups according to the acquisition sequence; and the weight value corresponding to the version number from high to low is the smallest, and the weight value corresponding to the version number is null.
Optionally, the consistency verification module further includes:
the switching verification unit is used for judging whether the monotonicity judgment group formed by the version numbers of the IO data corresponding to the data groups with the second number preset in the follow-up linked list to be verified is descending or horizontal; if yes, the data verification end overtime, and the error reporting operation is executed.
The invention also provides a data consistency verification system, which comprises: a data writing end and a data verification end as described above; wherein,,
the data writing end is used for distributing a disk for each linked list to be verified and recording metadata information of the linked list to be verified in the disk; and the metadata information comprises a version number of the linked list to be verified.
Optionally, the data writing end is further configured to generate a data set of the linked list to be verified at intervals of a predetermined time period.
The present invention also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the data consistency verification method as described above.
The invention provides a data consistency verification method, which comprises the following steps: acquiring a data group of a linked list to be verified of a data writing end; sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not; if the version number of the current IO data is incorrect, sequentially acquiring version numbers corresponding to the follow-up preset first number of IO data of the current IO data to form a monotonicity judging group; if the monotonicity judging group is in a first monotonicity state, the data writing end is slow, the data verification end is fast, and the current IO data verification is passed; if the monotonicity judging group is in the second monotonicity state, the data writing end is fast, the data verifying end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished; if the monotonicity judging group is in the third monotonic state, IO data is lost, and error reporting operation is executed.
Therefore, the method allocates version numbers to each linked list to be verified and corresponding IO data, and judges whether the data of the current data group of the data writing end and the data verification end are consistent or not by judging the monotonicity of a monotonicity judging group formed by the version numbers of the IO data of the linked list to be verified, so that whether the data of the current data group verified by the current data writing end and the data verification end are data obtained by synchronous communication or not is not required to be concerned; and the consistency verification of the data writing end and the data verification end under the condition of out-of-step is realized. The invention also provides a data consistency verification system, a data verification terminal and a computer readable storage medium, which have the beneficial effects and are not repeated here.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a data consistency verification method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a data verification terminal according to an embodiment of the present invention;
fig. 3 is a block diagram of a data consistency verification system according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, fig. 1 is a flowchart of a data consistency verification method according to an embodiment of the present invention; the method may include:
s110, acquiring a data group of a linked list to be verified of a data writing end;
the execution subject of the present embodiment is a data verification terminal. The present embodiment is not limited to the data verification terminal. For example, the disaster recovery physical machine can be used. The data verification terminal can deploy a plurality of examples at the same time, and each example can correspond to one data writing terminal; and the consistency verification of the data of a plurality of data writing ends is realized.
In this embodiment, the size of the data in the data set of the linked list to be verified is not limited. For example, the data group formed by IO data corresponding to the linked list to be verified in a preset time can be used. The embodiment does not limit the predetermined time, and the user may set the predetermined time according to his actual situation. For example 5 seconds. In this embodiment, in order to form the monotonicity determination group later, a version number needs to be set for the linked list to be verified and the corresponding data group. The present embodiment is not limited to a specific manner of setting the version number. In the following, a specific form is provided, and the execution body may be a program (for example, deployed at the data writing end (for example, deployed inside the production virtual machine, where each test virtual machine is deployed at most one)) inside the data writing end (the data writing end is not limited to the data writing end in this embodiment:
at the data writing end, space of one or more magnetic disks is allocated to one or more linked lists (i.e. linked lists to be verified) in units of blocks (minimum storage and processing units in a database), each linked list can be composed of a plurality of IO (Input/Output), and the space of the linked list spans across a plurality of magnetic disks. For subsequent formation of the monotonicity determination group, at this time, each IO records main information of a preset first number (the preset first number is not limited in this embodiment, for example, 4) of subsequent IOs, and the number of subsequent IOs is less than the preset first number and may be replaced by a null pointer. An IO is pre-assigned a corresponding preset first number of IOs before writing. Each disk has a fixed storage area for recording the same superblock, and the superblock records metadata information of all the linked lists, wherein the metadata information comprises version numbers of the linked lists and version numbers corresponding to IO data. The present embodiment does not limit the size and location of the fixed storage area. For example the first 4K of each disk. When all the linked list is written, a new linked list is regenerated (the new linked list has a new version number), the super block is updated and written from the beginning, the writing progress does not need to be related to the verification progress, and when the writing speed is higher than the verification speed, the rear-end collision phenomenon finally occurs. Each round of writing generates a new superblock, a superblock version number is generated, and the version number of each round is different, for example, the version number is monotonically increased or monotonically decreased every round. Typically monotonically increasing per round as the version number. The superblock version number is also included in the metadata of each IO and IO linked list to confirm that the read data is written in this round.
Further, the data writing end can generate a data group of the linked list to be verified every preset time period. For example, every 5 seconds, a data set corresponding to each linked list to be verified is formed. For example, the Qemu layer (analog processor layer) will be referred to as a point of a fractional data set every 5 seconds. A point pair of a fractional data set contains IOs for multiple disks within 5 seconds, which form a consistency group. The consistency group is used for guaranteeing consistency of recovery states of all the magnetic disks. A consistency group contains the data sets corresponding to the 5 seconds for each linked list to be verified.
The number of linked lists to be verified of the data writing end acquired at the same time is not limited in this embodiment. In this embodiment, a list to be verified is taken as an example, and when there are multiple lists to be verified, a verification process of the list to be verified may be referred to. For example, the data verification terminal may allocate a linked list verification object to each data writing terminal, for performing a data consistency verification process of the linked list to be verified. When a plurality of linked lists to be verified exist, a linked list verification object can be allocated to each linked list to be verified, and the linked list verification object is used for executing a data consistency verification process of the linked list to be verified. The specific data consistency verification logic is as follows.
S120, sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not;
in this embodiment, for each data group of the linked list to be verified, the IO data of the data group is sequentially obtained from the head of the linked list to be verified, the currently obtained IO data is used as the current IO data for subsequent verification, and if the verification is passed, the current IO data is verified, and the next IO data of the data group is continuously obtained for verification. First, whether the version number of the current IO data is correct or not is judged. Specifically, whether the version number of the current IO data is consistent with the version number of the linked list to be verified is judged. If the version numbers of the current IO data are consistent, the version numbers of the current IO data are correct, and if the version numbers of the current IO data are inconsistent, the version numbers of the current IO data are incorrect.
In the field of data processing and data communication, the sum of a group of data items used for verification purposes can be numbers or other character strings regarded as numbers in the process of computing the verification sum, and the relationship among a plurality of blocks is correct, the relationship with adjacent IOs is correct, and the like. The present embodiment is not limited thereto.
S130, if the version number of the current IO data is correct, the current IO data passes verification;
s140, if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the follow-up preset first number of IO data of the current IO data to form a monotonicity judging group;
specifically, if the version number of the current IO data is correct, the current IO data passes verification, and the next IO data is continuously read. If the version number of the current IO data is incorrect, version numbers corresponding to the follow-up preset first number of IO data of the current IO data are sequentially obtained to form a monotonicity judging group.
The embodiment does not limit the manner of forming the monotonicity determination group according to the version numbers corresponding to the preset first number of the subsequent IO data which sequentially acquire the current IO data. For example, the monotonicity determination groups can be directly formed according to the acquisition sequence by the version numbers acquired in sequence; or acquiring corresponding values according to the version numbers, and forming monotonicity judgment groups according to the acquired values and the acquisition sequence; and then, or according to the weight values corresponding to the version numbers acquired in sequence, the weight values are combined into a monotonicity judging group according to the acquisition sequence. The manner in which the monotonic decision group is determined at this time affects the specific form of the subsequent first, second and third monotonic states. For example, when the higher the version number is, the larger the corresponding value is, or the higher the version height is, the larger the corresponding weight value is, the first monotonic state is monotonically decreasing or horizontal, the second monotonic state is monotonically increasing, and the third monotonic state is decreasing followed by increasing.
Typically, the values corresponding to the version numbers are all sequentially increased. Optionally, sequentially obtaining version numbers corresponding to a preset first number of IO data of the current IO data to form the monotonicity determination group may include:
sequentially acquiring version numbers corresponding to the follow-up preset first number of IO data of the current IO data, and determining weight values corresponding to the version numbers;
forming monotonicity judgment groups by the weight values according to the acquisition sequence; and the weight value corresponding to the version number from high to low is the smallest, and the weight value corresponding to the version number is null.
Specifically, when the IO data with the correct version number cannot be obtained (if the IO data is empty or the old version number is read), the position information of the first number of IO data is preset from the follow-up of the current IO data in the data set, and the reading is attempted. The read results are sorted by weight as follows, and the weight values are sequentially reduced according to the following sequence: higher version number; equal version numbers; an old version number; cannot be read; empty (subsequent IO less than 4). The method is characterized in that the method is unreadable, the current version number is smaller than the current version number and equal to the current version number, a weight value is respectively set larger than the current version number, a monotonicity judging group is formed by each weight value according to the acquisition sequence, and the monotonicity of the monotonicity judging group can be obtained by observing the waveform change of the value in the monotonicity judging group.
S150, judging the monotonicity of the monotonicity judging group;
s160, if the monotonicity judging group is in a first monotonicity state, the data writing end is slow, the data verification end is fast, and the current IO data verification is passed;
specifically, if the monotonicity determination array is in a first monotonic state (e.g., monotonically decreasing or horizontal), it indicates that the data is not running down, and the current IO data is verified. But continue to determine if the subsequent data write end can catch up with the data verification end. If the user cannot catch up, the data verification end has overtime. Specifically, the data writing end can judge whether the data writing end can catch up with the data verification end by switching the points of the data grouping to try to read again at the same position. The present embodiment is not limited to a specific process. Optionally, after the current IO data is verified, the method may further include:
judging whether a monotonicity judging group is in a first monotonicity state or not formed by version numbers of IO data corresponding to a second number of data groups preset in the follow-up linked list to be verified;
if yes, the data verification end overtime, and executes error reporting operation.
Specifically, when the data writing end is slow and the data verifying end is fast, the data verifying end can detect the first monotonic state (such as monotonic decrease or monotonic level) of the monotonic judging group, at this time, a new data group is formed by switching points of the data groups and waiting for the data writing end, as long as the data writing end catches up with the data verifying end in the process that the data writing end switches the data groups of a subsequent preset second number, the verifying process can be continued normally. If the user cannot catch up, the data verification end overtime and reports errors, and at the moment, the data writing end is halted and exits. In this embodiment, the specific value of the preset second number is not limited, and may be, for example, 100, that is, whether the data verification end is to be traced by the subsequent 100 data sets is determined, that is, whether the monotonicity determining set no longer has the first monotonicity state.
The embodiment is not limited to the specific content of the specific data verification end timeout error-reporting operation. So long as the corresponding technician knows that the data verification end is overtime at the moment. The user can set specific content of the overtime error reporting operation of the data verification terminal according to actual conditions.
Further, when switching points of the sub-data sets, synchronization must be maintained between the multiple linked lists to be verified, that is, when each linked list to be verified initiates a point request for switching the sub-data sets, the points of the sub-data sets can be uniformly switched.
S170, if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verification end is slow, verification of the linked list to be verified passes, and verification of the linked list to be verified is finished;
specifically, when the data writing end is fast and the data verifying end is slow, a rear-end collision occurs, and the monotonic state (for example, monotonic rising) of the monotonic judging group occurs. And after the second monotone state is detected, directly finishing the verification of the linked list to be verified, and judging that the linked list passes. The super block is reread and the next round of verification is started.
S180, if the monotonicity judging group is in a third monotonicity state, IO data is lost, and error reporting operation is executed.
The embodiment is not limited to specific contents for executing the error reporting operation when the IO data is lost. So long as the corresponding technician knows that the IO data is lost at the moment. The user can set specific content of the overtime error reporting operation of the data verification terminal according to actual conditions. Such as which IO data comprising which linked list to be verified may be lost.
And after all the linked lists to be verified obtained from the data verification terminal pass verification, restarting reading the super block, and performing a new round of verification.
Based on the technical scheme, the data consistency verification method provided by the embodiment of the invention allocates version numbers to each linked list to be verified and corresponding IO data, judges whether the data of the current data group of the data writing end and the data verification end are consistent or not by judging the monotonicity of a monotonicity judging group formed by the version numbers of the IO data of the linked list to be verified, and does not need to care whether the data of the current data writing end and the data verification end verified data group are data acquired by synchronous communication or not; and the consistency verification of the data writing end and the data verification end under the condition of out-of-step is realized.
The data consistency verification system, the data verification terminal and the computer readable storage medium provided by the embodiments of the present invention are described below, and the data consistency verification system, the data verification terminal and the computer readable storage medium described below and the data consistency verification method described above can be referred to correspondingly.
Referring to fig. 2, fig. 2 is a block diagram of a data verification terminal according to an embodiment of the present invention; the data verification terminal may include:
an obtaining module 110, configured to obtain a data set of a linked list to be verified of a data writing end;
the version number verification module 120 is configured to sequentially obtain IO data of the data set, and determine whether the version number of the current IO data is correct;
the monotonicity judging group obtaining module 130 is configured to sequentially obtain version numbers corresponding to a preset first number of subsequent IO data of the current IO data to form a monotonicity judging group if the version number of the current IO data is incorrect;
the consistency verification module 140 is configured to, if the monotonicity determination group is in the first monotonic state, slow the data writing end, fast the data verification end, and pass the current IO data verification; if the monotonicity judging group is in the second monotonicity state, the data writing end is fast, the data verifying end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished; if the monotonicity judging group is in the third monotonic state, IO data is lost, and error reporting operation is executed.
Based on the above embodiment, the version number verification module 120 specifically acquires the IO data of the data set in sequence, and determines whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
Based on any of the above embodiments, the monotonicity determination group acquisition module 130 may include:
the weight value acquisition unit is used for sequentially acquiring version numbers corresponding to the follow-up preset first number IO data of the current IO data and determining weight values corresponding to the version numbers;
the monotonicity judging group acquisition unit is used for forming the weight values into monotonicity judging groups according to the acquisition sequence; and the weight value corresponding to the version number from high to low is the smallest, and the weight value corresponding to the version number is null.
Based on any of the above embodiments, the consistency verification module 140 may further include:
the switching verification unit is used for judging whether the monotonic decrease or the monotonic level of the monotonic level judgment group is formed by the version numbers of the IO data corresponding to the data groups of the second quantity preset in the follow-up linked list to be verified; if yes, the data verification end overtime, and executes error reporting operation.
It should be noted that, based on any embodiment, any module or unit may be implemented based on a programmable logic device, where the programmable logic device includes an FPGA, a CPLD, a single chip microcomputer, and the like.
Referring to fig. 3, fig. 3 is a block diagram illustrating a data consistency verification system according to an embodiment of the present invention; the system may include: a data writing end 200 and a data verification end 100 according to any of the above embodiments;
the data writing-in end 200 is used for distributing a disk for each linked list to be verified and recording metadata information of the linked list to be verified in the disk; the metadata information comprises a version number of a linked list to be verified.
Specifically, one linked list to be verified can be allocated to a plurality of disks, and a plurality of linked lists to be verified can be allocated to one disk. Each linked list to be verified can be composed of a plurality of IO data, and the space of the linked list can span a plurality of disks.
Based on the above embodiment, the data writing terminal 200 is further configured to generate a data set of the linked list to be verified at intervals of a predetermined period.
The present invention also provides a computer readable storage medium having a computer program stored thereon, which when executed by a processor implements the steps of the data consistency verification method according to any of the embodiments described above.
The computer readable storage medium may include: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In the description, each embodiment is described in a progressive manner, and each embodiment is mainly described by the differences from other embodiments, so that the same similar parts among the embodiments are mutually referred. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The data consistency verification method, the system, the data verification terminal and the computer readable storage medium provided by the invention are described in detail. The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to facilitate an understanding of the method of the present invention and its core ideas. It should be noted that it will be apparent to those skilled in the art that various changes and modifications can be made herein without departing from the principles of the invention, which are also intended to fall within the scope of the appended claims.
Claims (11)
1. A method of verifying data consistency, the method comprising:
acquiring a data group of a linked list to be verified of a data writing end;
sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not;
if the version number of the current IO data is incorrect, forming a monotonicity judging group according to the weight values corresponding to the version numbers acquired in sequence and the acquisition sequence; the monotonicity of the monotonicity judging group comprises a first monotonic state, a second monotonic state and a third monotonic state, wherein the first monotonic state is monotonically descending or horizontal, the second monotonic state is monotonically ascending, and the third monotonic state is descending first and then ascending;
if the monotonicity judging group is in the first monotonicity state, the data writing end is slow, the data verifying end is fast, and the current IO data is verified;
if the monotonicity judging group is in the second monotonicity state, the data writing end is fast, the data verification end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished;
and if the monotonicity judging group is in the third monotonicity state, IO data is lost, and error reporting operation is executed.
2. The method of claim 1, wherein determining whether the version number of the current IO data is correct comprises:
and judging whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
3. The method according to claim 2, wherein the grouping the weight values into the monotonicity determination group according to the sequentially acquired weight values corresponding to the version numbers includes:
the weight value corresponding to the version number from high to low is the smallest, and the weight value corresponding to the version number is null.
4. A method according to any one of claims 1-3, wherein after the current IO data verification passes, further comprising:
judging whether a monotonicity judging group is formed by version numbers of IO data corresponding to a second number of data groups preset in the follow-up linked list to be verified or not;
if yes, the data verification end overtime, and the error reporting operation is executed.
5. A data verification terminal, comprising:
the acquisition module is used for acquiring the data group of the linked list to be verified of the data writing end;
the version number verification module is used for sequentially acquiring the IO data of the data group and judging whether the version number of the current IO data is correct or not;
the monotonicity judging group acquisition module is used for forming a monotonicity judging group according to weight values corresponding to the version numbers acquired in sequence if the version numbers of the current IO data are incorrect; the monotonicity of the monotonicity judging group comprises a first monotonic state, a second monotonic state and a third monotonic state, wherein the first monotonic state is monotonically descending or horizontal, the second monotonic state is monotonically ascending, and the third monotonic state is descending first and then ascending;
the consistency verification module is used for enabling the data writing end to be slow and the data verification end to be fast if the monotonicity judging group is in the first monotonicity state, and enabling the current IO data to pass verification; if the monotonicity judging group is in the second monotonicity state, the data writing end is fast, the data verification end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished; and if the monotonicity judging group is in the third monotonicity state, IO data is lost, and error reporting operation is executed.
6. The data verification terminal according to claim 5, wherein the version number verification module is specifically configured to sequentially obtain the IO data of the data set, and determine whether a version number of the current IO data is consistent with a version number of the linked list to be verified.
7. The data verification terminal according to claim 6, wherein the monotonicity determination group acquisition module comprises:
and the monotonicity judging group acquisition unit is used for acquiring a weight value corresponding to a version number from high to low, wherein the weight value corresponding to a version number from high to low is the smallest, and the version number is null.
8. The data verification terminal according to any one of claims 5 to 7, wherein the consistency verification module further comprises:
the switching verification unit is used for judging whether the monotonicity judgment group formed by the version numbers of the IO data corresponding to the follow-up preset second number of data groups of the chain table to be verified is in the first monotonicity state or not; if yes, the data verification end overtime, and the error reporting operation is executed.
9. A data consistency verification system, the system comprising: a data writing end and a data verification end according to any one of claims 5-8; wherein,,
the data writing end is used for distributing a disk for each linked list to be verified and recording metadata information of the linked list to be verified in the disk; and the metadata information comprises a version number of the linked list to be verified.
10. The system of claim 9, wherein the data writing side is further configured to generate the data set of the linked list to be verified every predetermined time period.
11. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the data consistency verification method according to any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810410346.7A CN110442474B (en) | 2018-05-02 | 2018-05-02 | Data consistency verification method, system and data verification terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810410346.7A CN110442474B (en) | 2018-05-02 | 2018-05-02 | Data consistency verification method, system and data verification terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110442474A CN110442474A (en) | 2019-11-12 |
| CN110442474B true CN110442474B (en) | 2023-07-11 |
Family
ID=68427204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810410346.7A Active CN110442474B (en) | 2018-05-02 | 2018-05-02 | Data consistency verification method, system and data verification terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110442474B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111291371A (en) * | 2020-01-10 | 2020-06-16 | 北京深之度科技有限公司 | Application program security verification method and device |
| CN116182931B (en) * | 2023-04-26 | 2023-07-14 | 苏州英瑞传感技术有限公司 | Consistency evaluation method, controller and monitoring system of experimental production verification equipment |
| CN117539705B (en) * | 2024-01-10 | 2024-06-11 | 深圳鲲云信息科技有限公司 | Verification method, device and system of system on chip and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104714859A (en) * | 2013-12-17 | 2015-06-17 | 南京壹进制信息技术有限公司 | Rapid backup and recovery method of mass files |
| CN107861832A (en) * | 2017-09-27 | 2018-03-30 | 深信服科技股份有限公司 | Data verification method, device and readable storage medium storing program for executing |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8140909B2 (en) * | 2008-02-22 | 2012-03-20 | International Business Machines Corporation | Efficient method to detect disk write errors |
| US7908512B2 (en) * | 2008-03-05 | 2011-03-15 | International Business Machines Corporation | Method and system for cache-based dropped write protection in data storage systems |
| US9792191B2 (en) * | 2015-08-19 | 2017-10-17 | Nxp Usa, Inc. | Fast write mechanism for emulated electrically erasable (EEE) system |
-
2018
- 2018-05-02 CN CN201810410346.7A patent/CN110442474B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104714859A (en) * | 2013-12-17 | 2015-06-17 | 南京壹进制信息技术有限公司 | Rapid backup and recovery method of mass files |
| CN107861832A (en) * | 2017-09-27 | 2018-03-30 | 深信服科技股份有限公司 | Data verification method, device and readable storage medium storing program for executing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110442474A (en) | 2019-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442474B (en) | Data consistency verification method, system and data verification terminal | |
| CN112214369A (en) | Hard disk fault prediction model establishing method based on model fusion and application thereof | |
| EP3629182B1 (en) | Generating a test script execution order | |
| US8396840B1 (en) | System and method for targeted consistency improvement in a distributed storage system | |
| TW201232294A (en) | Formatting data by example | |
| CN110008129B (en) | Reliability test method, device and equipment for storage timing snapshot | |
| CN109684140A (en) | A kind of slow disk detection method, device, equipment and computer readable storage medium | |
| CN108875061A (en) | A kind of conformance test method and relevant apparatus of distributed file system | |
| US20160335555A1 (en) | Making a prediction regarding development of a software product | |
| CN108021509B (en) | Test case dynamic sequencing method based on program behavior network aggregation | |
| CN110347590A (en) | The interface testing control method and device of operation system | |
| CN111176885A (en) | Data verification method and related device for distributed storage system | |
| CN102736932B (en) | The method for making of image file, image file and the starting factor method of multisystem | |
| CN108717385A (en) | A kind of data reconstruction method and system for flash memory | |
| CN112000513A (en) | Computer and VPD data operation method, device and storage medium thereof | |
| CN114116355A (en) | Memory test method and device and electronic equipment | |
| CN111078790B (en) | Method and system for synchronizing isolated block data in block chain and storage medium | |
| CN115878052A (en) | RAID array inspection method, inspection device and electronic equipment | |
| CN107229858B (en) | PECI bus master-slave machine verification system and method supporting crc8 verification | |
| CN109359093A (en) | A kind of rule file update method and system | |
| CN109032962B (en) | Data storage method and device and embedded system | |
| CN114594973A (en) | Chip firmware burning method, device and system and storage medium | |
| CN113342651A (en) | Recovery method for testing case defect and case fuzzy association relation | |
| CN116955030B (en) | Test plan distribution method and system, electronic device and storage medium | |
| US11822829B2 (en) | Method, device, and program product for data rebuilding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |