CN112416891B

CN112416891B - Data detection method, device, electronic equipment and readable storage medium

Info

Publication number: CN112416891B
Application number: CN202011366737.7A
Authority: CN
Inventors: 张朝潞
Original assignee: Beijing Topsec Technology Co Ltd; Beijing Topsec Network Security Technology Co Ltd; Beijing Topsec Software Co Ltd
Current assignee: Beijing Topsec Technology Co Ltd; Beijing Topsec Network Security Technology Co Ltd; Beijing Topsec Software Co Ltd
Priority date: 2020-11-26
Filing date: 2020-11-26
Publication date: 2023-11-28
Anticipated expiration: 2040-11-26
Also published as: CN112416891A

Abstract

The application provides a data detection method, a data detection device, electronic equipment and a readable storage medium, and relates to the technical field of computer data processing. The method comprises the following steps: acquiring a first check value of a first copy of the target file set and a second check value of a second copy of the target file set, wherein the first check value is a check value determined based on a check algorithm and check values of all first data blocks in the first copy, and the second check value is a check value determined based on the check algorithm and check values of all second data blocks in the second copy; according to the first check value and the second check value, whether the first copy and the second copy have abnormal detection results is determined, so that whether the first copy and the second copy have abnormal detection can be rapidly realized, and the operation amount in the detection process is reduced.

Description

Data detection method, device, electronic equipment and readable storage medium

Technical Field

The present application relates to the field of computer data processing, and in particular, to a data detection method, apparatus, electronic device, and readable storage medium.

Background

In a distributed storage system, data redundancy is usually implemented in the form of multiple copies, and when an abnormality occurs in a disk or a host where a copy exists, other copies can still provide complete data. If silent data damage occurs, namely, data is required to be processed by various software and hardware in the transmission process of the distributed storage system, errors occur in any processing process, so that errors occur in the data content, and inconsistent situation occurs in multiple copies of data in the distributed storage system. Wherein each copy typically includes a plurality of data blocks, each data block including data content and a check value. At present, the mode of judging whether the copy is abnormal is as follows: reading each data block in each stored copy, performing operation based on the data content in the read data block to obtain a new check value, and comparing the new check value with the check value carried by the data block, wherein the check value carried by the data block can be the check value obtained by performing operation on the data content in the data block before storage, and in the storage process, the data content in the data block can be changed, so that the check values obtained by successive operation can be different. The operation amount of the method is large, and the distributed storage system cannot quickly realize copy detection.

Disclosure of Invention

An embodiment of the application aims to provide a data detection method, a data detection device, electronic equipment and a readable storage medium, which can solve the problems of large operand and low detection speed in the detection process of whether copies of a target file set are abnormal or not.

In order to achieve the above object, an embodiment of the present application is achieved by:

in a first aspect, an embodiment of the present application provides a data detection method, where the method includes:

acquiring a first check value of a first copy of a target file set and a second check value of a second copy of the target file set, wherein the first check value is a check value determined based on a check algorithm and check values of all first data blocks in the first copy, and the second check value is a check value determined based on the check algorithm and check values of all second data blocks in the second copy;

and determining whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value.

In the above embodiment, for the first copy and the second copy of the target file set, the check values of the first copy and the second copy may be directly used for comparison, if the data of the first copy and the second copy have no error, the check values of the first copy and the second copy are the same, and if the data of the first copy and the second copy have error, the check values of the first copy and the second copy are different, so that whether the first copy and the second copy have an abnormality can be quickly detected, which is beneficial to reducing the operation amount in the detection process.

With reference to the first aspect, in some alternative embodiments,

the first copy includes a plurality of first components, each of the plurality of first components including a plurality of first data blocks, the second copy includes a plurality of second components, each of the plurality of second components including a plurality of second data blocks;

before obtaining the first check value of the first copy of the target file set and the second check value of the second copy of the target file set, the method further includes:

for each first component in the plurality of first components, calculating the check value of all first data blocks in each first component through the check algorithm to obtain a check value representing each first component, wherein the check value of the first data blocks is as follows: a check value obtained by calculating the first data in the stored first data block through the check algorithm;

calculating the check values corresponding to the plurality of first components through the check algorithm to obtain the first check value representing the first copy;

for each second component in the plurality of second components, calculating the check value of all second data blocks in each second component through the check algorithm to obtain a check value representing each second component, wherein the check value of each second data block is as follows: calculating a check value obtained by the second data in the stored second data block through the check algorithm;

And operating the check values corresponding to the second components through the check algorithm to obtain the second check value representing the second copy.

In the above embodiment, for the data structure of the copy of the target file set, the check value of the component is calculated from the check value of the data block at the bottom layer, and then the check value of the copy is calculated using the check value of the component. Therefore, the check value of the copy can be obtained quickly without operating the data in the data block, so that the check value of the obtained copy is used as the basis for detecting whether the copy is abnormal or not, and the quick detection of the copy is realized.

With reference to the first aspect, in some optional embodiments, the method further includes:

when the detection result represents that the first copy and/or the second copy are abnormal, comparing the check value of each first component with the check value of a corresponding second component in the plurality of second components;

if the check value of the first component and the check value of the second component of any group which are compared are different, the first component in any group is taken as a first target component, and the second component in any group is taken as a second target component;

Comparing the check value of each first data block in the first target assembly with the check value of the corresponding second data block in the second target assembly;

if the check value of the first data block and the check value of the second data block of any group which are compared are different, the first data block and the second data block in any group are determined to be abnormal data blocks.

In the above embodiment, if the check values of the first copy and the second copy are abnormal, the data block indicating that there is an abnormality in the underlying data block of at least one of the first copy and the second copy. The check values of the first data block and the second data block in any group are different, namely the data block with the abnormality. In this way, the location where the abnormal data block exists can be quickly determined from the first copy and the second copy.

With reference to the first aspect, in some optional embodiments, the first copy includes a plurality of first data blocks, and the second copy includes a plurality of second data blocks;

Calculating the check values of the plurality of first data blocks in the first copy through the check algorithm to obtain and store the first check values representing the first copy, wherein the check values of the first data blocks are as follows: a check value obtained by calculating the first data in the stored first data block through the check algorithm;

calculating the check values of the plurality of second data blocks in the second copy through the check algorithm to obtain and store the second check values representing the second copy, wherein the check values of the second data blocks are as follows: and calculating the check value obtained by the operation of the second data in the stored second data block through the check algorithm.

In the above embodiments, there may be no need for a component between the copy and the data block. The check value of each data block in a single copy can be calculated, so that the check value of the copy can be obtained quickly, and the subsequent quick detection of the copy based on the check value of the copy is facilitated.

when the detection result represents that the first copy and/or the second copy are abnormal, comparing the check value of each first data block with the check value of the corresponding second data block in the second copy;

obtaining check values of target data blocks corresponding to the first data blocks or the second data blocks in any group in all other copies of the target file set, wherein all other copies are copies except the first copy and the second copy in the target file set;

determining a standard data block representing normal data from the first data block, the second data block and all the target data blocks in any group according to the check value of the first data block in any group, the check value of the second data block in any group and the check values of all the target data blocks, wherein the check values of the standard data block are as follows: and in the first data block, the second data block and all the target data blocks in any group, when the same check values exist, the check value corresponding to the maximum number of the same check values is used as the check value of the standard data block for representing the normal data.

In the above embodiment, after determining the abnormal first data block and the abnormal second data block, the standard data blocks corresponding to the first data block and the second data block and representing the normal data may be determined by combining the data blocks corresponding to the other copies of the target file set, so as to replace the abnormal data blocks with the standard data blocks.

and replacing the data blocks with check values different from those of the standard data blocks in the first data block and the second data block in any group by the standard data blocks.

In the above-described embodiment, the validity of the stored data can be improved by replacing the abnormal data block with the standard data block.

With reference to the first aspect, in some optional embodiments, determining whether the first copy and the second copy have an abnormal detection result according to the first check value and the second check value includes:

when the first check value is the same as the second check value, determining and obtaining a detection result that no abnormality exists in the first copy and the second copy;

and when the first check value is different from the second check value, determining and obtaining a detection result of the first copy and/or the second copy with abnormality.

With reference to the first aspect, in some optional embodiments, the target file set is stored in a distributed storage system.

In a second aspect, an embodiment of the present application further provides a data detection apparatus, where the apparatus includes:

the system comprises an acquisition unit, a verification unit and a storage unit, wherein the acquisition unit is used for acquiring a first verification value of a first copy of a target file set and a second verification value of a second copy of the target file set, the first verification value is determined based on a verification algorithm and verification values of all first data blocks in the first copy, and the second verification value is determined based on the verification algorithm and verification values of all second data blocks in the second copy;

and the detection determining unit is used for determining whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value.

In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a processor and a memory coupled to each other, where the memory stores a computer program, and when the computer program is executed by the processor, causes the electronic device to perform the method described above.

In a fourth aspect, embodiments of the present application also provide a computer-readable storage medium having stored therein a computer program which, when run on a computer, causes the computer to perform the above-described method.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Fig. 2 is a flow chart of a data detection method according to an embodiment of the present application.

Fig. 3 is a schematic diagram of a target data set according to an embodiment of the present application.

Fig. 4 is a schematic structural diagram of a data unit in a target data set according to an embodiment of the present application.

FIG. 5 is a second schematic diagram of a target data set according to an embodiment of the present application.

Fig. 6 is a block diagram of a data detection device according to an embodiment of the present application.

Icon: 10-an electronic device; 11-a processing module; 12-a memory module; 13-a communication module; 110-an acquisition unit; 120-a detection determination unit.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. It should be noted that the terms "first," "second," and the like are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.

In view of the above problems, the applicant of the present application proposes the following examples to solve the above problems. The following embodiments and features of the embodiments may be combined with each other without conflict.

Referring to fig. 1, an embodiment of the present application provides an electronic device 10, which can quickly detect each redundant copy of a target file set in a distributed storage system to determine whether the redundant copy of the target file set is abnormal. The target file set is the set of files to be detected. The volume of the target file set is typically large and may include one or more files. In other embodiments, the target file set may be data in one logical storage volume of a distributed storage system, where the logical storage volume is well known to those skilled in the art.

In a distributed storage system, for the same target file set, there are typically at least two copies corresponding to the target file set, each of which may be a complete target file set. Each copy typically includes a plurality of data blocks that may be stored in a distributed storage system in a distributed manner within a corresponding server, in a manner well known to those skilled in the art. If the data of all the copies in the same target file set are normal, the data content of each copy in the same target file set is the same; if the data of any copy is abnormal, the data content of each copy in the same target file set is different. Wherein the stored data block comprises data content and a check value. The check value may be a value calculated by a check algorithm based on the data content in the data block before distributed storage. The verification algorithm includes, but is not limited to, a CRC (Cyclic Redundancy Check ) algorithm, a Hash (Hash) algorithm, and the like.

The electronic device 10 may be any server in a distributed storage system, or the electronic device 10 may be a network device independent of the distributed storage system, where the network device may be a personal computer, a server, or the like, and the type of the electronic device 10 is not specifically limited herein.

In this embodiment, the electronic device 10 may include a processing module 11 and a storage module 12. The memory module 12 stores a computer program which, when executed by said processing module 11, enables the electronic device 10 to perform the steps of the method described below.

Of course, the electronic device 10 may also include other modules, for example, the electronic device 10 may also include a communication module 13 for establishing a communication connection with the distributed storage system for data interaction with the distributed storage system. For example, the electronic device 10 may obtain the check value in the data block from a distributed storage system. In addition, the electronic device 10 may further include a data detection apparatus 100 solidified in the memory module 12.

The processing module 11, the storage module 12, the communication module 13 and the various elements of the data detection device 100 are electrically connected directly or indirectly to achieve data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.

Referring to fig. 2, the embodiment of the present application further provides a data detection method, which can be applied to the above-mentioned electronic device 10, and the steps of the method are executed or implemented by the electronic device 10. The method may comprise the steps of:

Step S210, obtaining a first check value of a first copy of a target file set and a second check value of a second copy of the target file set, wherein the first check value is a check value determined based on a check algorithm and check values of all first data blocks in the first copy, and the second check value is a check value determined based on the check algorithm and check values of all second data blocks in the second copy;

step S210, determining whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value.

The steps of the method will be described in detail below, as follows:

in step S210, the target file set may be stored in a distributed storage system, or the target file set may be stored in a stand-alone storage device. Wherein the target file set typically includes at least two copies. In the case that no anomaly or corruption of the data of a single copy occurs, the data of each copy is identical to the data of one complete set of target files. The first copy in the target set of files may refer to one copy or a class of copies. The second copy in the target set of files may be one copy or a class of copies.

For example, the first copy of the target set of files is one copy, and the second copy is a type of copy, e.g., the second copy may be the remaining copy of the target set of files except for the first copy. Alternatively, the first copy and the second copy are any two copies in the target file set, respectively. The meaning of the first copy and the second copy may be interpreted according to actual situations, and is not specifically limited herein.

The first check value is the check value of the first copy, and if the number of the first copies is multiple, the first check value comprises the check values of the multiple copies. Similarly, the second check value is the check value of the second copy. If the number of the second copies is plural, the second check value includes check values of plural copies. Wherein the first check value and the second check value have been stored in a distributed storage system or a stand-alone storage device before performing step S210. Thus, in performing step S210, the electronic device 10 may obtain the first check value of the first copy and the second check value of the second copy from the distributed storage system or the stand-alone storage device.

In step S220, the electronic device 10 may quickly determine whether the data of the first copy and the second copy are abnormal based on the obtained first check value and the second check value.

In this embodiment, step S220 may include:

It is understood that if the data of the first copy and the second copy are not in error, the check values of the first copy and the second copy are the same. If the data of the first copy and the second copy are in error, the check values of the first copy and the second copy are typically different. Based on this, the electronic device 10 may quickly determine whether the first copy and the second copy are abnormal by comparing the first check value of the first copy and the second check value of the second copy without performing other check operations on the data blocks in the first copy and the second copy.

When the number of copies of the target file set is plural and the first copy is any one of the plural copies and the second copy is the rest of the plural copies except the first copy, the second check value includes a check value of the rest of the copies except the first copy. That is, the second check value includes a plurality of check values. At this time, when determining the detection result, step S220 may include:

And when the same check values exist in the first check values of the first copies and the second check values of all the second copies, the check value corresponding to the most number of the same check values is used as the check value for representing the normal of the data, any copy corresponding to the check value for representing the normal of the data is determined as a standard copy, and the copy with the check value different from the check value of the standard copy is determined as an abnormal copy. And then, carrying out layer-by-layer comparison on the check value based on the data units of each level in the abnormal copy and the check value based on the data units of each level in the standard copy so as to locate the abnormal data block.

Referring to fig. 3, as an alternative embodiment, the first copy includes a plurality of first components, each of the plurality of first components includes a plurality of first data blocks, and the second copy includes a plurality of second components, each of the plurality of second components includes a plurality of second data blocks. In other implementations, the set of destination files may be represented by Volume; copies may be represented by Replica; the Component may be represented by Component and the data block may be represented by Chunk. Prior to step S210, the method may further include:

In this embodiment, the components and data blocks can be regarded as data units of different capacity levels. The capacities of the data units of different magnitudes can be set according to practical situations. For example, the data size of one component may be 1GB and the data size of one data block may be 2MB.

For each copy of the same target file set, if the data is not wrong, the data sizes of components among the copies are the same, and the data sizes of data blocks in the components are the same. In storing duplicate data, a distributed storage system may divide a copy of a target set of files into multiple components, and divide each component into multiple blocks of data, and then store the blocks of data in a distributed manner as is well known to those skilled in the art. For example, for data units of the same magnitude, if the data size of the divided data does not reach the set size of the data unit, data can be padded by filling data so that the data sizes of the data units of the same magnitude are all the same.

After each data block of the copy is stored in a corresponding disk block in the distributed storage system in a scattered manner, the data content of the data block can be operated by a verification algorithm based on the content of the stored data block, and the obtained value is the verification value of the data block and is added in the data block. In general, the check values of data blocks having different data contents are different, and the check values of data blocks having the same data contents are the same. The operation mode for obtaining the check value by operating the data content of the data block according to the check algorithm is well known to those skilled in the art. For example, the CRC algorithm is used to perform CRC operation on the data content of the data block, and the obtained value is the check value of the data block. Or, carrying out Hash operation on the data content of the data block by utilizing a Hash algorithm, wherein the obtained Hash value is the check value of the data block. After the data content of the data block is calculated, the structure of the obtained data block may be as shown in fig. 4.

Referring to fig. 4, the structure of the data unit may be a duplicate structure, a component structure, or a data block structure. For example, the data unit is a data block, and one data block includes a data area and a check area. The data in the data area is the data content which needs to be operated by the checking algorithm. The data of the check area may include T, R, P three parts of data. The P-part data may represent a check value obtained by performing the above-described operation on the data content of the data area, and the T-part data may be a time stamp for generating the check value. The R part data can represent a read check mark and can be used as a mark bit of a check area. For example, the data content stored in the data area of the first data block is the first data; the data content stored in the data area of the second data block is the second data.

After the check values of all the first data blocks in the first copy are obtained through operation, the check values of all the first data blocks in each first component can be operated through a check algorithm, and the obtained values are the check values of the corresponding components. And then, calculating the check values in all the first components through a check algorithm, wherein the obtained value is the first check value of the first copy. The second check value of the second copy operates in a similar manner to the first check value, and will not be described in detail herein.

In the process of obtaining the first check value and the second check value through operation, the check value of the data block is directly utilized to operate because the operation is not needed for the data area in each data block, so that the operation amount is reduced, and the check value of the first copy is obtained quickly.

In the process of obtaining the first check value and the second check value through operation, the adopted check algorithms are the same, for example, both are CRC algorithms or both are Hash algorithms.

As an alternative embodiment, the method may further comprise the step of locating the abnormal data block. For example, the method may further comprise:

In this embodiment, each first component of the first copy may be provided with a corresponding component number, where the component numbers of different first components are different, so as to distinguish the different first components; the first data blocks in each first component may be provided with corresponding data block numbers, the data block numbers of different first data blocks in the same first component being different for distinguishing between different first data blocks. Similarly, in the second copy, the first component corresponding to the first copy may be provided with the same or a matching or associated component number; in each second component, each second data block may be provided with a data block number that is the same as or matches a first data block in a corresponding first component in the first copy. For each copy of the same target file set, all data blocks with the same component number and the same data block number can be indexed based on the data block number and the component number; all components with the same component number may also be indexed. In the case that no error occurs in the data, the check values of all the data blocks with the same component number and the same data block number are the same, and the check values of all the components with the same component number are the same.

It will be appreciated that if the first check value and the second check value are different, it means that there is an anomaly in at least one of the first copy and the second copy, that is, in the first copy and the second copy, there is at least one group of first components and second components whose check values are different in the first component and the second component that are matched (e.g., component numbers are the same). Among the first and second components having different check values, there is at least one set of first and second data blocks having different check values among the first and second data blocks that are matched (e.g., component numbers are the same and data block numbers are the same). For any group of matched first components and second components, if the check values of the components are the same, the data blocks in the components do not need to be compared.

In the above embodiment, the larger data unit with larger capacity is screened to determine the larger data unit where the abnormal data is located, and then the smaller data units in the larger data unit are compared and screened, so that all the smaller data units do not need to be compared, and the calculation amount can be reduced and the minimum data unit with the abnormality can be rapidly located through multi-layer screening.

Referring to fig. 5, as an alternative embodiment, the first copy includes a plurality of first data blocks, and the second copy includes a plurality of second data blocks. That is, there may be no data units on the order of components between the copy and the data block. Before step S210, the manner of calculating the check value of the copy may be as follows, that is, before step S210, the method may further include:

It can be understood that when a data unit of the order of magnitude of a component does not exist between the copy and the data block, when the first check value of the first copy is determined, the check values of all the first data blocks in the first copy can be directly operated through a check algorithm, and the obtained value is the first check value, so that the check value of the first component does not need to be determined, and the first check value of the first copy can be obtained quickly. The operation manner of the second check value of the second copy is similar to that of the first check value, and will not be described herein.

As an alternative embodiment, the method may further comprise:

When locating the abnormal data block based on the data structure shown in fig. 5, the check values of each group of the first data block and the second data block that are matched can be compared until all the data blocks in the first copy and the second copy are traversed. Wherein, the matched first data block and the second data block are understood as the same or the numbers of the data blocks are associated. And under the condition that the data are normal, the check values of the first data block and the second data block in any matched group are the same. If the data is abnormal, at least one group of matched first data block and second data block exist, and the check values of the data blocks are different.

The first data block and the second data block with different check values are abnormal data blocks, and subsequent data correction is needed.

When the number of copies of the target file set is multiple and the first copy and the second copy are any two copies of the multiple copies, as an alternative embodiment, the method may further include:

Wherein the first data block and the second data block in any group can be regarded as: any one group of first data block and second data block with the same number of data blocks (and the same component number).

It will be appreciated that for the same target set of files, in each copy, if there is a data error, typically in the same number of data blocks (e.g., the same component number and the same data block number), the data in which there is a small number of data blocks is erroneous or abnormal, but still some data blocks have the same data content, the parity value of the data blocks whose data content is the same, and the same number of the parity values that characterize the data as normal is typically the largest. Based on the method, in the data blocks with the same number, the check values of the data blocks are analyzed and compared to judge whether the same check value exists, and then the number of the same check values is counted to obtain the ratio of the number of the different check values to the total number of the data blocks with the same number. The check value corresponding to the maximum number ratio is the check value representing the normal data, and any data block corresponding to the check value can be used as a standard data block.

For example, the number of copies of the same target file set is 6, the number of data blocks with the same number is usually 6 (if there is missing data, the number is less than 6), and among the 6 data blocks, if one data block has a check value of M, where two data blocks have check values of N, and the remaining three data blocks have check values of Q, the number of check values of Q is the largest, any data block corresponding to the check value of Q is normal data, and any data block corresponding to the check value of Q can be used as a standard data block. Wherein M, N, Q is assumed to represent different check values, respectively, for ease of understanding the scheme.

As an alternative implementation manner, when the number of the copies of the target file set is multiple, and the first copy is any one copy of the multiple copies, and the second copy is the rest copy except the first copy in the multiple copies, at this time, in the matched first data block and second data block, the check value of the second data block includes the check values of multiple data blocks with the same number. Among the data blocks with different check values, the mode of determining the standard data block for representing the normal data can be as follows:

And taking any data block corresponding to the check value representing the normal data as the standard data block when the same check value exists in the check values of the first data block and the check values of all the second data blocks in any group.

As an alternative embodiment, the method may further comprise:

In the above embodiment, in any of the matched data blocks, if the check value of the current data block is different from the check value of the standard data block of the matched data block, it indicates that there is an abnormality or an error in the current data block, and correction is required. At this time, the abnormal data block can be replaced by the standard data block of the group, so that the abnormal data block can be corrected, and the validity of the stored data can be improved.

It should be noted that, in other embodiments, the hierarchical structure of the data units included in the copy may be different from that shown in fig. 3 or fig. 5. The copies may include more orders of magnitude of data units. For example, a type of data unit may be added between the copies and components of fig. 5, e.g., the data unit is referred to as a data set, i.e., one copy may include multiple data sets and one data set includes multiple components. When abnormal data is located, in the embodiment, check values of large-capacity data units are compared firstly to screen out large-capacity data units with different check values; and then comparing the check values of the data units with smaller capacity in the data units with larger capacity and with different check values, and determining the data units with different check values from the data units with the smallest capacity by layer-by-layer comparison. The data units with the same check value do not need to be analyzed and compared, so that all the data units with smaller capacity do not need to be analyzed and processed, the operation amount is reduced, and the positioning of abnormal data is realized quickly.

Referring to fig. 6, an embodiment of the present application further provides a data detection apparatus 100, which can be applied to the above-mentioned electronic device 10, for executing each step in the method. The data detection device 100 includes at least one software functional module that may be stored in the memory module 12 in the form of software or Firmware (Firmware) or cured in an Operating System (OS) of the electronic device 10. The processing module 11 is configured to execute executable modules stored in the storage module 12, such as software functional modules and computer programs included in the data detection device 100.

In this embodiment, the data detection apparatus 100 may include the acquisition unit 110 and the detection determination unit 120, and the following operations may be performed:

an obtaining unit 110, configured to obtain a first check value of a first copy of a target file set and a second check value of a second copy of the target file set, where the first check value is a check value determined based on a check algorithm and check values of all first data blocks in the first copy, and the second check value is a check value determined based on the check algorithm and check values of all second data blocks in the second copy;

The detection determining unit 120 is configured to determine whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value.

Optionally, the data detection device 100 may further include a check value determination unit. The first copy includes a plurality of first components, each of the plurality of first components includes a plurality of first data blocks, and the second copy includes a plurality of second components, each of the plurality of second components includes a plurality of second data blocks. Before the acquisition unit 110 performs step S210, the check value determination unit is configured to:

Optionally, the data detection device 100 may further include an alignment determination unit for:

Optionally, the first copy includes a plurality of first data blocks, and the second copy includes a plurality of second data blocks. Before the acquisition unit 110 performs step S210, the check value determination unit is configured to:

Optionally, the alignment determination unit may be further configured to: when the detection result represents that the first copy and/or the second copy are abnormal, comparing the check value of each first data block with the check value of the corresponding second data block in the second copy; if the check value of the first data block and the check value of the second data block of any group which are compared are different, the first data block and the second data block in any group are determined to be abnormal data blocks.

Alternatively, the acquisition unit 110 may be further configured to: and acquiring check values of target data blocks corresponding to the first data blocks or the second data blocks in any group in all other copies of the target file set, wherein all other copies are copies except the first copy and the second copy in the target file set.

The alignment determination unit may also be configured to: determining a standard data block representing normal data from the first data block, the second data block and all the target data blocks in any group according to the check value of the first data block in any group, the check value of the second data block in any group and the check values of all the target data blocks, wherein the check values of the standard data block are as follows: and in the first data block, the second data block and all the target data blocks in any group, when the same check values exist, the check value corresponding to the maximum number of the same check values is used as the check value of the standard data block for representing the normal data.

Optionally, the data detecting apparatus 100 may further include a data correction unit for replacing, by the standard data block, a data block in which a check value in the first data block and the second data block in the arbitrary group is different from a check value of the standard data block.

Alternatively, the detection determination unit 120 may be further configured to:

In this embodiment, the processing module 11 may be an integrated circuit chip with signal processing capability. The processing module 11 may be a general purpose processor. For example, the processor may be a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), a Field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present application.

The memory module 12 may be, but is not limited to, random access memory, read only memory, programmable read only memory, erasable programmable read only memory, electrically erasable programmable read only memory, and the like. In this embodiment, the storage module 12 may be configured to store check values and the like for each copy of the target file set. Of course, the storage module 12 may also be used to store a program, which is executed by the processing module 11 upon receiving an execution instruction.

It is to be understood that the configuration shown in fig. 1 is merely a schematic diagram of one configuration of the electronic device 10, and that the electronic device 10 may include more components than those shown in fig. 1. The components shown in fig. 1 may be implemented in hardware, software, or a combination thereof.

It should be noted that, for convenience and brevity of description, specific working processes of the electronic device 10 and the data detection apparatus 100 described above may refer to corresponding processes of each step in the foregoing method, and will not be described in detail herein.

The embodiment of the application also provides a computer readable storage medium. The computer-readable storage medium has stored therein a computer program which, when run on a computer, causes the computer to perform the data detection method as described in the above embodiments.

From the foregoing description of the embodiments, it will be apparent to those skilled in the art that the present application may be implemented in hardware, or by means of software plus a necessary general hardware platform, and based on this understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disc, a mobile hard disk, etc.), and includes several instructions for causing a computer device (may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective implementation scenario of the present application.

In summary, the embodiments of the present application provide a data detection method, apparatus, electronic device, and readable storage medium. The method comprises the following steps: acquiring a first check value of a first copy of the target file set and a second check value of a second copy of the target file set, wherein the first check value is a check value determined based on a check algorithm and check values of all first data blocks in the first copy, and the second check value is a check value determined based on the check algorithm and check values of all second data blocks in the second copy; and determining whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value. In the scheme, aiming at the first copy and the second copy of the target file set, check values of the first copy and the second copy can be directly utilized for comparison, if the data of the first copy and the second copy are not wrong, the check values of the first copy and the second copy are identical, and if the data of the first copy and the second copy are wrong, the check values of the first copy and the second copy are different, so that whether the first copy and the second copy have abnormality detection can be rapidly realized, and the operation amount in the detection process is reduced.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus, system and method may be implemented in other manners as well. The above-described apparatus, system, and method embodiments are merely illustrative, for example, flow charts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of data detection, the method comprising:

determining whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value;

when the first copy comprises a plurality of first components, each first component in the plurality of first components comprises a plurality of first data blocks, the second copy comprises a plurality of second components, and each second component in the plurality of second components comprises a plurality of second data blocks, when the detection result indicates that the first copy and/or the second copy are abnormal, the check value of each first component is compared with the check value of the corresponding second component in the plurality of second components;

if the check value of the first data block and the check value of the second data block of any group which are compared are different, determining the first data block and the second data block in any group as abnormal data blocks;

determining a standard data block representing normal data from the first data block, the second data block and all the target data blocks in any group according to the check value of the first data block in any group, the check value of the second data block in any group and the check values of all the target data blocks, wherein the check values of the standard data block are as follows: in the first data block, the second data block and all the target data blocks in any group, when the same check values exist, the check value corresponding to the maximum number of the same check values is used as the check value of the standard data block representing that the data are normal;

2. The method of claim 1, wherein, in the case where the first copy includes a plurality of first components, each of the plurality of first components includes a plurality of first data blocks, the second copy includes a plurality of second components, each of the plurality of second components includes a plurality of second data blocks, the method further comprises, prior to obtaining the first check value of the first copy of the target file set and the second check value of the second copy of the target file set:

3. The method of claim 1, wherein, in the case where the first copy includes only a plurality of first data blocks and the second copy includes only a plurality of second data blocks, before obtaining the first check value of the first copy of the target file set and the second check value of the second copy of the target file set, the method further comprises:

4. A method according to claim 3, wherein in case the first copy comprises only a plurality of first data blocks and the second copy comprises only a plurality of second data blocks, the method further comprises:

5. The method of claim 1, wherein determining whether the first copy and the second copy have an abnormal detection result according to the first check value and the second check value comprises:

6. The method of claim 1, wherein the set of target files is stored in a distributed storage system.

7. A data detection device, the device comprising:

the detection determining unit is used for determining whether the first copy and the second copy have abnormal detection results according to the first check value and the second check value;

A comparison determining unit, configured to, when the first copy includes a plurality of first components, each of the plurality of first components includes a plurality of first data blocks, the second copy includes a plurality of second components, and each of the plurality of second components includes a plurality of second data blocks, compare a check value of each first component with a check value of a corresponding second component of the plurality of second components when the detection result indicates that there is an abnormality in the first copy and/or the second copy; if the check value of the first component and the check value of the second component of any group which are compared are different, the first component in any group is taken as a first target component, and the second component in any group is taken as a second target component; comparing the check value of each first data block in the first target assembly with the check value of the corresponding second data block in the second target assembly; if the check value of the first data block and the check value of the second data block of any group which are compared are different, determining the first data block and the second data block in any group as abnormal data blocks; the method comprises the steps of,

and the data correction unit is used for replacing the data blocks with the check values different from the check values of the standard data blocks in the first data block and the second data block in any group through the standard data blocks.

8. An electronic device comprising a processor and a memory coupled to each other, the memory storing a computer program that, when executed by the processor, causes the electronic device to perform the method of any of claims 1-6.

9. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein a computer program which, when run on a computer, causes the computer to perform the method according to any of claims 1-6.