CN110442474A - A kind of data consistency checking method, system and data verification end - Google Patents
A kind of data consistency checking method, system and data verification end Download PDFInfo
- Publication number
- CN110442474A CN110442474A CN201810410346.7A CN201810410346A CN110442474A CN 110442474 A CN110442474 A CN 110442474A CN 201810410346 A CN201810410346 A CN 201810410346A CN 110442474 A CN110442474 A CN 110442474A
- Authority
- CN
- China
- Prior art keywords
- data
- monotonicity
- verification
- group
- verified
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013524 data verification Methods 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000012795 verification Methods 0.000 claims description 84
- 238000004590 computer program Methods 0.000 claims description 3
- 238000013502 data validation Methods 0.000 claims 1
- 238000010200 validation analysis Methods 0.000 claims 1
- 230000015572 biosynthetic process Effects 0.000 abstract 1
- 230000008569 process Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000003247 decreasing effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000011084 recovery Methods 0.000 description 4
- 230000001360 synchronised effect Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1479—Generic software techniques for error detection or fault masking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
- G06F11/2082—Data synchronisation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses data consistency checking method, system, data verification end and computer readable storage medium, method includes obtaining the data group of the chained list to be verified at data write-in end；The I/O data for successively obtaining data group, judges whether the version number of current I/O data is correct；If incorrect, the subsequent corresponding version number of the first quantity I O data that presets for successively obtaining current I/O data forms monotonicity decision set；If monotonicity decision set is the first Monotonic state, current I/O data is verified；If the second Monotonic state, chained list to be verified is verified, and terminates the verifying of chained list to be verified；If third Monotonic state, there are I/O data loss, execute the operation that reports an error；The monotonicity that this method passes through the monotonicity decision set of I/O data formation for treating verifying chained list is determined can determine whether the data of the data write-in end data group current with data verifying end data are consistent, realize the consistency checking under data write-in end and data verifying end step loss condition.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method and a system for verifying data consistency, a data verification end, and a computer-readable storage medium.
Background
The consistency check is an important means for ensuring the reliability of data, and at present, a write-in end and a verification end are deployed in the same machine in the existing consistency check scheme, so that the synchronization of the write-in end and the verification end is easy to realize, and the verification is realized on the premise that a data chain table is complete. The data link table must be complete is a prerequisite for the normal operation of the existing consistency verification scheme, i.e. the synchronization of the write end and the verification end must be ensured. However, in many application scenarios, synchronization between the write end and the verification end is difficult to guarantee. For example, in a disaster recovery system, a production end and a disaster recovery end are deployed in different devices across regions and connected through a high-latency network. The write-in end and the verification end lack a real-time communication mechanism, so that the write-in end and the verification end are asynchronous, the write-in end and the verification end are fast and slow, and a data linked list is partially covered and incomplete; the former is slow and the latter is fast, the write end can not catch up with the verification end in time of downloading the disk, and invalid data can be read. This results in the inability of existing consistency verification schemes to work effectively.
Therefore, how to implement data consistency check under the condition that the write end and the verification end are not synchronous is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
The invention aims to provide a data consistency verification method, a data consistency verification system, a data verification end and a computer readable storage medium, which can realize consistency verification under the condition that a data writing end and the data verification end are out of step.
In order to solve the above technical problem, the present invention provides a data consistency verification method, including:
acquiring a data group of a linked list to be verified of a data write-in end;
sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not;
if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the subsequent preset first amount of IO data of the current IO data to form a monotonicity judging group;
if the monotonicity determination group is in a first monotonicity state, the data writing end is slow, the data verification end is fast, and the current IO data verification is passed;
if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the linked list to be verified passes verification, and the verification of the linked list to be verified is finished;
and if the monotonicity judging group is in a third monotonicity state, IO data loss exists, and error reporting operation is executed.
Optionally, the determining whether the version number of the current IO data is correct includes:
and judging whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
Optionally, sequentially obtaining version numbers corresponding to subsequent preset first amount of IO data of the current IO data to form a monotonicity determination group, including:
sequentially acquiring version numbers corresponding to subsequent preset first number of IO data of the current IO data, and determining a weight value corresponding to each version number;
forming a monotonicity judgment group by the weight values according to an acquisition sequence; and the weight value corresponding to the version number from high to low is the minimum weight value corresponding to the version number in a null state.
Optionally, after the current IO data passes the verification, the method further includes:
judging whether a monotonicity judging group formed by the version numbers of IO data corresponding to the subsequent preset second quantity of data groups of the linked list to be verified is in the first monotonicity state;
if yes, the data verification end is overtime, and the error reporting operation is executed.
The invention also provides a data verification terminal, comprising:
the acquisition module is used for acquiring a data group of a linked list to be verified of the data write-in end;
the version number verification module is used for sequentially acquiring IO data of the data group and judging whether the version number of the current IO data is correct or not;
a monotonicity determination group obtaining module, configured to, if the version number of the current IO data is incorrect, sequentially obtain version numbers corresponding to subsequent preset first number of IO data of the current IO data to form a monotonicity determination group;
the consistency verification module is used for enabling the data write end to be slow and the data verification end to be fast if the monotonicity judgment group is in a first monotonicity state, and enabling the current IO data to pass verification; if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the linked list to be verified passes verification, and the verification of the linked list to be verified is finished; and if the monotonicity judging group is in a third monotonicity state, IO data loss exists, and error reporting operation is executed.
Optionally, the version number verification module specifically obtains the IO data of the data group in sequence, and determines whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
Optionally, the monotonicity determination group obtaining module includes:
the weight value acquiring unit is used for sequentially acquiring version numbers corresponding to subsequent preset first number of IO data of the current IO data and determining the weight value corresponding to each version number;
a monotonicity decision group acquisition unit for forming a monotonicity decision group by the weight values in an acquisition order; and the weight value corresponding to the version number from high to low is the minimum weight value corresponding to the version number in a null state.
Optionally, the consistency verification module further includes:
the switching verification unit is used for judging whether a monotonicity judgment group is monotonically decreased or horizontal or not formed by the version numbers of the IO data corresponding to the subsequent preset second quantity of data groups of the linked list to be verified; if yes, the data verification end is overtime, and the error reporting operation is executed.
The invention also provides a data consistency verification system, which comprises: a data write end and a data verification end as described above; wherein,
the data write-in end is used for distributing a disk for each linked list to be verified and recording the metadata information of the linked list to be verified in the disk; wherein, the metadata information includes the version number of the linked list to be verified.
Optionally, the data write-in terminal is further configured to generate a data group of the linked list to be verified at predetermined time intervals.
The present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the data consistency verification method as described above.
The invention provides a data consistency verification method, which comprises the following steps: acquiring a data group of a linked list to be verified of a data write-in end; sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not; if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the subsequent preset first amount of IO data of the current IO data to form a monotonicity judging group; if the monotonicity judging group is in a first monotonicity state, the data writing end is slow, the data verification end is fast, and the current IO data verification is passed; if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the linked list to be verified passes verification, and the verification of the linked list to be verified is finished; if the monotonicity determination group is in the third monotonicity state, IO data loss exists, and error reporting operation is executed.
The method can be seen in that a version number is distributed to each chain table to be verified and corresponding IO data, whether the data of the current data group of the data writing end and the data verifying end are consistent can be judged by judging the monotonicity of a monotonicity judging group formed by the version numbers of the IO data of the chain tables to be verified, and whether the data of the data group verified by the current data writing end and the data verifying end are data acquired by synchronous communication is not needed to be concerned; and realizing consistency verification under the condition that the data writing end and the data verification end are out of step. The invention also provides a data consistency verification system, a data verification end and a computer readable storage medium, which have the beneficial effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a flow chart of a data consistency verification method according to an embodiment of the present invention;
fig. 2 is a block diagram of a data verification end according to an embodiment of the present invention;
fig. 3 is a block diagram of a data consistency verification system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating a data consistency verification method according to an embodiment of the present invention; the method can comprise the following steps:
s110, acquiring a data group of a linked list to be verified of a data write-in end;
the execution subject of this embodiment is the data verification side. The present embodiment does not limit the data verification end. For example, it may be a disaster recovery end physical machine. The data verification end can simultaneously deploy a plurality of instances, and each instance can correspond to one data write end; and the consistency verification of the data of a plurality of data writing ends is realized.
In this embodiment, the size of the data in the data group of the linked list to be verified is not limited. For example, the data group may be formed by IO data corresponding to the linked list to be verified within a predetermined time. The preset time is not limited in this embodiment, and the user may set the preset time according to the actual situation of the user. For example 5 seconds. In this embodiment, in order to form a monotonicity determination group subsequently, version numbers need to be set for the linked list to be verified and the corresponding data group. The embodiment does not limit the specific setting manner of the version number. A specific form is provided below, where the execution body may be a program inside a data write terminal (the embodiment does not limit the data write terminal; for example, the execution body may be a production-side virtual machine) (for example, the execution body is deployed inside the data write terminal (for example, the execution body is deployed inside the production-side virtual machine, and at most one execution body is deployed for each test virtual machine)):
at the data write end, allocating space of one or more disks to one or more linked lists (i.e., to-be-verified linked lists) in units of Block (the minimum storage and processing unit in the database), where each linked list may be composed of multiple IO (Input/Output), and the space of the linked list spans multiple disks. In order to form the monotonicity determination set subsequently, at this time, each IO records main information of a preset first number (the preset first number is not limited in this embodiment, for example, 4) of subsequent IOs, and the number of the subsequent IOs is less than the preset first number and may be replaced by a null pointer. The associated preset first number of IOs is pre-allocated before an IO is written. Each disk has a fixed storage area for recording the same superblock, and metadata information of all linked lists is recorded in the superblock, including the version number of the linked list and the version number corresponding to IO data. The size and location of the fixed storage area are not limited in this embodiment. For example the first 4K of each disk. When all the linked lists are written in, the new linked list is regenerated again (the new linked list has a new version number), the super block is updated and is written in from the head, the writing progress does not need to be related to the verification progress, and when the writing speed is higher than the verification speed, the rear-end collision phenomenon can occur finally. The superblock version number is generated every write round, and the version number of each write round is different, for example, the version number is monotonically increased or monotonically decreased every write round. Typically such as a version number that monotonically increases for each round. The superblock version number is also included in the metadata of each IO and IO chain to confirm that the read data was written in this round.
Further, the data write-in terminal may generate the data group of the linked list to be verified at predetermined time intervals. For example, every 5 seconds, a data set corresponding to each linked list to be verified is formed. For example, the Qemu layer (analog processor layer) will be referred to as a point of one fractional data set every 5 seconds. The 5 seconds of a point pair of a partial data group contain IO to multiple disks that make up a consistency group. The consistency group is used for ensuring consistency of recovery states of the disks. A consistency group contains the 5 second data sets for each linked list to be verified.
The present embodiment does not limit the number of linked lists to be verified of the data write ends acquired at the same time. In this embodiment, one linked list to be verified is taken as an example for explanation, and when a plurality of linked lists to be verified exist, the verification process of one linked list to be verified may be referred to. For example, the data verification end may allocate a linked list verification object to each data write end, so as to execute a data consistency verification process corresponding to the linked list to be verified. When a plurality of linked lists to be verified exist, a linked list verification object can be distributed to each linked list to be verified and used for executing the data consistency verification process of the corresponding linked list to be verified. The specific data consistency verification logic follows the following steps.
S120, sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not;
in this embodiment, for each data group of the linked list to be verified, the IO data of the data group is sequentially acquired from the head of the linked list to be verified, and the currently acquired IO data is used as the current IO data for subsequent verification. Firstly, whether the version number of the current IO data is correct is judged. Specifically, whether the version number of the current IO data is consistent with the version number of the linked list to be verified is judged. And if the version numbers of the current IO data are consistent, the version number of the current IO data is correct, and if the version numbers of the current IO data are inconsistent, the version number of the current IO data is incorrect.
Further, to improve the accuracy of the IO data, the correctness of the IO data may also be verified at this time, for example, the checksum (computer sum check code, checksum. in the data processing and data communication fields, the sum of a set of data items for verification purpose, which may be numbers or other character strings regarded as numbers in the process of calculating the checksum) within the IO is verified to be correct, the relationship between a plurality of blocks is correct, the relationship with the adjacent IO is correct, and the like. This embodiment is not limited to this.
S130, if the version number of the current IO data is correct, the current IO data passes verification;
s140, if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the subsequent preset first amount of IO data of the current IO data to form a monotonicity judging group;
specifically, if the version number of the current IO data is correct, the current IO data passes verification, and the next IO data continues to be read. And if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the subsequent preset first amount of IO data of the current IO data to form a monotonicity judging group.
The present embodiment does not limit the manner of forming the monotonicity determination set according to the version numbers corresponding to the subsequent preset first amount of IO data of the current IO data obtained in sequence. For example, monotonicity determination groups may be directly formed in the order of acquisition with sequentially acquired version numbers; or acquiring corresponding numerical values according to the version number, and forming a monotonicity determination group by the acquired numerical values according to the acquisition sequence; and then or according to the weight values corresponding to the sequentially acquired version numbers, forming a monotonicity judgment group by the weight values according to the acquisition sequence. The manner in which the monotonicity decision set is determined at this time affects the specific form of the subsequent first, second and third monotonicity states. For example, when the version number is higher and the corresponding value is larger, or the version height is higher and the corresponding weight value is larger, the first monotonic state is monotonically decreased or horizontal, the second monotonic state is monotonically increased, and the third monotonic state is first decreased and then increased.
Usually, the values corresponding to the version numbers are sequentially increased. Optionally, sequentially obtaining version numbers corresponding to subsequent preset first amount of IO data of the current IO data to form a monotonicity determination group may include:
sequentially acquiring version numbers corresponding to subsequent preset first number of IO data of the current IO data, and determining a weight value corresponding to each version number;
forming a monotonicity judgment group by the weight values according to the acquisition sequence; and the weight value corresponding to the version number from high to low is the minimum weight value corresponding to the version number in a null state.
Specifically, when IO data with a correct version number cannot be acquired (for example, a null or an old version number is read), position information of subsequent preset first amount of IO data of current IO data in the data group is read and attempted to be read. The read results are sorted according to the weight, and the weight values are reduced in the following order: a higher version number; an equal version number; an old version number; cannot be read; null (less than 4 subsequent IO). The monotonicity of the monotonicity decision group can be obtained by observing the waveform change of the value in the monotonicity decision group.
S150, judging the monotonicity of the monotonicity judging group;
s160, if the monotonicity determination group is in a first monotonicity state, the data writing end is slow, the data verification end is fast, and the current IO data verification is passed;
specifically, if the monotonicity determination array is in a first monotonicity state (e.g., monotonicity decreasing or level), it indicates that the data is not downloaded and the current IO data is verified. But it is continuously determined whether the subsequent data write side can catch up with the data verification side. If the data can not be traced, the data verification end is overtime. Specifically, the reading can be tried again at the same position by switching the point of the sub data group to determine whether the data write end can catch up with the data verification end. The present embodiment does not limit the specific process. Optionally, after the current IO data passes the verification, the method may further include:
judging whether a monotonicity judging group formed by the version numbers of IO data corresponding to the subsequent preset second quantity of data groups of the linked list to be verified is in a first monotonicity state;
if yes, the data verification end is overtime, and error reporting operation is executed.
Specifically, when the data write end is slow and the data verification end is fast, the data verification end can detect a first monotonicity state (for example, monotonicity reduction or level) of the monotonicity determination set, at this time, a new data set is formed by switching the points of the data set and waiting for the data write end, and as long as the data write end catches up with the data verification end in the process of switching the data verification end to a subsequent preset second number of data sets, the verification process can be continued normally. If the data can not be traced, the data verification end is overtime and reports an error, and the data writing end is halted and exits at the moment. In this embodiment, the specific value of the preset second number is not limited, and may be, for example, 100, that is, it is necessary to determine whether to catch up with the data verification end through the subsequent 100 data sets, that is, whether the monotonicity determination set no longer has the first monotonicity state.
The embodiment does not limit the specific content of the timeout error reporting operation of the data verification terminal. So long as corresponding technicians know that the timeout occurs at the data verification end. The user can set specific contents of the overtime error reporting operation of the data verification terminal according to actual conditions.
Further, when switching the point of the sub data group, the multiple linked lists to be verified must be kept synchronous, that is, when each linked list to be verified initiates a point request for switching the sub data group, the point of the sub data group can be uniformly switched.
S170, if the monotonicity determination group is in a second monotonicity state, the data writing end is fast, the data verification end is slow, the verification of the linked list to be verified is passed, and the verification of the linked list to be verified is finished;
specifically, when the data write end is fast, the data verification end is slow, a tail-end collision occurs, and the monotonicity determination set has a second monotonicity state (e.g., monotonicity rising). And when the second monotone state is detected, directly finishing the verification of the linked list to be verified, and concluding that the linked list to be verified passes. The superblock is reread and the next round of verification is started.
And S180, if the monotonicity determination group is in the third monotonicity state, IO data loss exists, and error reporting operation is executed.
The embodiment does not limit the specific content of performing the error reporting operation when there is IO data loss. So long as corresponding technicians know that the data is in the IO data loss state at this time. The user can set specific contents of the overtime error reporting operation of the data verification terminal according to actual conditions. For example, which IO data containing which to-be-verified linked list may be lost.
And when all the to-be-verified linked lists acquired from the data verification end pass the verification, reading the super blocks again, and performing a new round of verification.
Based on the technical scheme, the method for verifying the data consistency provided by the embodiment of the invention allocates a version number to each chain table to be verified and corresponding IO data, judges whether the data of the current data group of the data writing end and the data verification end are consistent by judging the monotonicity of a monotonicity judgment group formed by the version numbers of the IO data of the chain tables to be verified, and does not need to care whether the data of the data group verified by the current data writing end and the data verification end are the data acquired by synchronous communication; and realizing consistency verification under the condition that the data writing end and the data verification end are out of step.
The data consistency verification system, the data verification end and the computer readable storage medium described below may be referred to in correspondence with the data consistency verification method described above.
Referring to fig. 2, fig. 2 is a block diagram of a data verification end according to an embodiment of the present invention; the data verifying terminal may include:
an obtaining module 110, configured to obtain a data group of a linked list to be verified at a data write end;
a version number verification module 120, configured to sequentially obtain IO data of the data group, and determine whether a version number of the current IO data is correct;
a monotonicity determination group obtaining module 130, configured to, if the version number of the current IO data is incorrect, sequentially obtain version numbers corresponding to subsequent preset first amount of IO data of the current IO data to form a monotonicity determination group;
the consistency verification module 140 is configured to, if the monotonicity determination group is in the first monotonicity state, slow data write end, fast data verification end, and pass current IO data verification; if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the linked list to be verified passes verification, and the verification of the linked list to be verified is finished; if the monotonicity determination group is in the third monotonicity state, IO data loss exists, and error reporting operation is executed.
Based on the above embodiment, the version number verification module 120 specifically obtains the IO data of the data group in sequence, and determines whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
Based on any of the above embodiments, the monotonicity decision group obtaining module 130 may include:
the weight value acquiring unit is used for sequentially acquiring version numbers corresponding to subsequent preset first number of IO data of the current IO data and determining the weight value corresponding to each version number;
a monotonicity decision group acquisition unit for forming a monotonicity decision group by the weight values in an acquisition order; and the weight value corresponding to the version number from high to low is the minimum weight value corresponding to the version number in a null state.
Based on any of the above embodiments, the consistency verification module 140 may further include:
the switching verification unit is used for judging whether the monotonicity formed by the version numbers of the IO data corresponding to the data groups with the second quantity preset in the following of the linked list to be verified is monotonously reduced or horizontal; if yes, the data verification end is overtime, and error reporting operation is executed.
It should be noted that, based on any of the above embodiments, any module or unit may be implemented based on a programmable logic device, where the programmable logic device includes an FPGA, a CPLD, a single chip, and the like.
Referring to fig. 3, fig. 3 is a block diagram illustrating a data consistency verification system according to an embodiment of the present invention; the system may include: a data write terminal 200 and a data verification terminal 200 as described in any of the embodiments above; wherein,
a data write-in terminal 200, configured to allocate a disk for each linked list to be verified, and record metadata information of the linked list to be verified in the disk; wherein, the metadata information comprises the version number of the linked list to be verified.
Specifically, one linked list to be verified may be allocated to multiple disks, and multiple linked lists to be verified may also be allocated to one disk. Each chain table to be verified can be composed of a plurality of IO data, and the space of the chain table can span a plurality of disks.
Based on the above embodiment, the data write terminal 200 is further configured to generate the data group of the linked list to be verified every predetermined time period.
The present invention also provides a computer readable storage medium having a computer program stored thereon, which, when being executed by a processor, implements the steps of the data consistency verification method according to any of the embodiments described above.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The data consistency verification method, system, data verification end and computer readable storage medium provided by the present invention are described in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (11)
1. A method for data consistency verification, the method comprising:
acquiring a data group of a linked list to be verified of a data write-in end;
sequentially acquiring IO data of the data group, and judging whether the version number of the current IO data is correct or not;
if the version number of the current IO data is incorrect, sequentially acquiring the version numbers corresponding to the subsequent preset first amount of IO data of the current IO data to form a monotonicity judging group;
if the monotonicity determination group is in a first monotonicity state, the data writing end is slow, the data verification end is fast, and the current IO data verification is passed;
if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the linked list to be verified passes verification, and the verification of the linked list to be verified is finished;
and if the monotonicity judging group is in a third monotonicity state, IO data loss exists, and error reporting operation is executed.
2. The method of claim 1, wherein determining whether the version number of the current IO data is correct comprises:
and judging whether the version number of the current IO data is consistent with the version number of the linked list to be verified.
3. The method according to claim 2, wherein sequentially obtaining version numbers corresponding to subsequent preset first amount of IO data of the current IO data to form a monotonicity decision group includes:
sequentially acquiring version numbers corresponding to subsequent preset first number of IO data of the current IO data, and determining a weight value corresponding to each version number;
forming a monotonicity judgment group by the weight values according to an acquisition sequence; and the weight value corresponding to the version number from high to low is the minimum weight value corresponding to the version number in a null state.
4. The method according to any one of claims 1 to 3, wherein after the current IO data verification passes, further comprising:
judging whether a monotonicity judging group formed by the version numbers of IO data corresponding to the subsequent preset second quantity of data groups of the linked list to be verified is in the first monotonicity state;
if yes, the data verification end is overtime, and the error reporting operation is executed.
5. A data verification end, comprising:
the acquisition module is used for acquiring a data group of a linked list to be verified of the data write-in end;
the version number verification module is used for sequentially acquiring IO data of the data group and judging whether the version number of the current IO data is correct or not;
a monotonicity determination group obtaining module, configured to, if the version number of the current IO data is incorrect, sequentially obtain version numbers corresponding to subsequent preset first number of IO data of the current IO data to form a monotonicity determination group;
the consistency verification module is used for enabling the data write end to be slow and the data verification end to be fast if the monotonicity judgment group is in a first monotonicity state, and enabling the current IO data to pass verification; if the monotonicity judging group is in a second monotonicity state, the data writing end is fast, the data verifying end is slow, the linked list to be verified passes verification, and the verification of the linked list to be verified is finished; and if the monotonicity judging group is in a third monotonicity state, IO data loss exists, and error reporting operation is executed.
6. The data verification end according to claim 5, wherein the version number verification module is specifically configured to sequentially obtain the IO data of the data group, and determine whether a version number of the current IO data is consistent with a version number of the linked list to be verified.
7. The data verification terminal according to claim 6, wherein the monotonicity decision group obtaining module comprises:
the weight value acquiring unit is used for sequentially acquiring version numbers corresponding to subsequent preset first number of IO data of the current IO data and determining the weight value corresponding to each version number;
a monotonicity decision group acquisition unit for forming a monotonicity decision group by the weight values in an acquisition order; and the weight value corresponding to the version number from high to low is the minimum weight value corresponding to the version number in a null state.
8. The data validation terminal of any one of claims 5-7, wherein the consistency validation module further comprises:
the switching verification unit is used for judging whether a monotonicity judgment group is formed by the version numbers of IO data corresponding to the subsequent preset second quantity of data groups of the linked list to be verified or not is in a first monotonicity state; if yes, the data verification end is overtime, and the error reporting operation is executed.
9. A data consistency verification system, characterized in that the system comprises: a data write terminal and a data verification terminal as claimed in any one of claims 5 to 8; wherein,
the data write-in end is used for distributing a disk for each linked list to be verified and recording the metadata information of the linked list to be verified in the disk; wherein, the metadata information includes the version number of the linked list to be verified.
10. The system according to claim 9, wherein the data write terminal is further configured to generate the data groups of the linked list to be verified at predetermined time intervals.
11. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the data consistency verification method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810410346.7A CN110442474B (en) | 2018-05-02 | 2018-05-02 | Data consistency verification method, system and data verification terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810410346.7A CN110442474B (en) | 2018-05-02 | 2018-05-02 | Data consistency verification method, system and data verification terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110442474A true CN110442474A (en) | 2019-11-12 |
| CN110442474B CN110442474B (en) | 2023-07-11 |
Family
ID=68427204
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810410346.7A Active CN110442474B (en) | 2018-05-02 | 2018-05-02 | Data consistency verification method, system and data verification terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110442474B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111291371A (en) * | 2020-01-10 | 2020-06-16 | 北京深之度科技有限公司 | Application program security verification method and device |
| CN116182931A (en) * | 2023-04-26 | 2023-05-30 | 苏州英瑞传感技术有限公司 | Consistency evaluation method, controller and monitoring system of experimental production verification equipment |
| CN117539705A (en) * | 2024-01-10 | 2024-02-09 | 深圳鲲云信息科技有限公司 | Verification method, device and system of system on chip and electronic equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090213487A1 (en) * | 2008-02-22 | 2009-08-27 | International Business Machines Corporation | Efficient method to detect disk write errors |
| US20090228744A1 (en) * | 2008-03-05 | 2009-09-10 | International Business Machines Corporation | Method and system for cache-based dropped write protection in data storage systems |
| CN104714859A (en) * | 2013-12-17 | 2015-06-17 | 南京壹进制信息技术有限公司 | Rapid backup and recovery method of mass files |
| US20170052859A1 (en) * | 2015-08-19 | 2017-02-23 | Freescale Semiconductor, Inc. | Fast write mechanism for emulated electrically erasable (eee) system |
| CN107861832A (en) * | 2017-09-27 | 2018-03-30 | 深信服科技股份有限公司 | Data verification method, device and readable storage medium storing program for executing |
-
2018
- 2018-05-02 CN CN201810410346.7A patent/CN110442474B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090213487A1 (en) * | 2008-02-22 | 2009-08-27 | International Business Machines Corporation | Efficient method to detect disk write errors |
| US20090228744A1 (en) * | 2008-03-05 | 2009-09-10 | International Business Machines Corporation | Method and system for cache-based dropped write protection in data storage systems |
| CN104714859A (en) * | 2013-12-17 | 2015-06-17 | 南京壹进制信息技术有限公司 | Rapid backup and recovery method of mass files |
| US20170052859A1 (en) * | 2015-08-19 | 2017-02-23 | Freescale Semiconductor, Inc. | Fast write mechanism for emulated electrically erasable (eee) system |
| CN107861832A (en) * | 2017-09-27 | 2018-03-30 | 深信服科技股份有限公司 | Data verification method, device and readable storage medium storing program for executing |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111291371A (en) * | 2020-01-10 | 2020-06-16 | 北京深之度科技有限公司 | Application program security verification method and device |
| CN116182931A (en) * | 2023-04-26 | 2023-05-30 | 苏州英瑞传感技术有限公司 | Consistency evaluation method, controller and monitoring system of experimental production verification equipment |
| CN117539705A (en) * | 2024-01-10 | 2024-02-09 | 深圳鲲云信息科技有限公司 | Verification method, device and system of system on chip and electronic equipment |
| CN117539705B (en) * | 2024-01-10 | 2024-06-11 | 深圳鲲云信息科技有限公司 | Verification method, device and system of system on chip and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110442474B (en) | 2023-07-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110442474A (en) | A kind of data consistency checking method, system and data verification end | |
| EP3506104A1 (en) | Coverage test support device and coverage test support method | |
| CN110008129B (en) | Reliability test method, device and equipment for storage timing snapshot | |
| CN112037058B (en) | Data verification method, device and storage medium | |
| CN108875061A (en) | A kind of conformance test method and relevant apparatus of distributed file system | |
| CN111176885A (en) | Data verification method and related device for distributed storage system | |
| CN115167891B (en) | Data updating method, device and equipment of interface control file and storage medium | |
| CN108154904A (en) | A kind of test method and device of storage chip performance | |
| CN113391767B (en) | Data consistency checking method and device, electronic equipment and readable storage medium | |
| CN111625400A (en) | Data redundancy processing method and device and storage medium | |
| CN112148218A (en) | Method, device and equipment for storing check data of disk array and storage medium | |
| CN111984421A (en) | Data processing method, device and storage medium | |
| CN106407123A (en) | Automated testing method and device of server interface | |
| CN114253920A (en) | Transaction reordering method, device, equipment and readable storage medium | |
| CN108196975B (en) | Data verification method and device based on multiple checksums and storage medium | |
| CN111444197B (en) | Verification method, device and equipment for data records in block chain type account book | |
| CN117541385A (en) | Frequent financial transaction mining method and device based on association rule item | |
| CN110990207B (en) | BPS memory test method, system, terminal and storage medium based on Whitley platform | |
| CN112416417A (en) | Code amount statistical method and device, electronic equipment and storage medium | |
| CN116955193A (en) | Interface testing method, device, equipment and storage medium | |
| CN112015558A (en) | Data verification method, device and storage medium | |
| CN106445737A (en) | Multi-backup starting method | |
| CN107392745B (en) | Reconciliation data fragmentation processing method | |
| CN110427528A (en) | SSD identifier test method, device, computer equipment and storage medium | |
| CN113326046B (en) | Method and device for acquiring compiling time length |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |