WO2017008677A1

WO2017008677A1 - Data detection method and device

Info

Publication number: WO2017008677A1
Application number: PCT/CN2016/089079
Authority: WO
Inventors: 梁永锋
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2015-07-16
Filing date: 2016-07-07
Publication date: 2017-01-19
Also published as: CN106649056B; CN106649056A

Abstract

Disclosed are a data detection method and device. The data detection method comprises: when synchronizing data of a specified file, acquiring parameter information about the synchronized data, wherein the parameter information is associated with the synchronized data (101); determining whether each piece of parameter information corresponding to the acquired specified file has a vacancy (102); and if so, determining that data loss occurs in the specified file (103). The detection method not only improves the detection accuracy for lost data, but also has a simple and feasible detection process and relatively high detection efficiency.

Description

Data detection method and device

The present application claims priority to Chinese Patent Application Serial No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Technical field

The present application relates to the field of data processing technologies, and in particular, to a data detection method and a data detection device.

Background technique

In various web applications, there are many log files that are constantly generated, such as a buried log on the page for recording user behavior. These logs are distributed across many servers and can have up to 10W servers with multiple log files on each machine. In the business, these log files need to be synchronized to the data warehouse system for analysis. Therefore, in the synchronization process, it is necessary to ensure that the data is not lost, so as to provide real guarantee for the service.

At present, in the process of synchronizing massive data, there are two schemes for detecting whether data is lost. One is a sample check, which periodically outputs a regular log in the log, and then detects whether these regular logs exist at the target location (data warehouse system). If these regular logs do not exist, the data is considered to be lost. The other is the final consistency check. In the data collection process, the last record number of the file is sent to the target location, and then the number of records finally received is counted at the target location. If the number of records is inconsistent, the data is considered to be lost.

However, for sampling inspection, due to the existence of a certain sampling rate, it is highly prone to data loss but cannot be detected. For the final consistency check, in the process of data transmission, data duplication often occurs. If there is no deduplication, then the final number of records counted in the target is inaccurate. It can be seen that the above detection method has low detection accuracy for lost data.

Therefore, a technical problem that needs to be solved urgently by those skilled in the art is how to improve the detection accuracy of lost data.

Summary of the invention

The technical problem to be solved by the embodiments of the present application is to provide a data detection method, which can improve the detection accuracy of lost data.

Correspondingly, the embodiment of the present application further provides a data detecting apparatus for ensuring implementation and application of the foregoing method.

In order to solve the above problems, the present application discloses a data detection method, including:

Obtaining parameter information of the synchronization data when the data of the specified file is synchronized, the parameter information being associated with the synchronization data;

Determining whether there is a gap in each parameter information corresponding to the specified file that has been obtained;

If yes, it is determined that there is data loss in the specified file.

Further, the determining whether the obtained parameter information corresponding to the specified file has a gap includes:

The parameter information of the acquired synchronization data is inserted into the parameter information linked list of the specified file, where the parameter information linked list includes parameter information corresponding to the specified file acquired earlier;

Merging adjacent parameter information in the parameter information linked list;

The parameter information of the vacant parameter information in the merged parameter information list is found according to the preset condition.

Further, searching for the parameter information of the vacant parameter information in the merged parameter information list according to the preset condition, including:

Determining, by using a predetermined number of thresholds, whether there is a parameter information of the vacancy in the merged parameter information list. When the search result is the parameter information of the vacancy and the number of times of searching reaches the threshold number, the determined location is determined. There is a gap in the parameter information corresponding to the specified file.

Further, the method further includes:

Determining, by the merged data corresponding to the parameter information of the parameter information list, the data of the specified file is lost.

Further, the parameter information of the synchronization data includes an offset and a data length, and the offset of the next synchronization data is the sum of the offset of the synchronization data and the data length.

The embodiment of the present application further discloses a data detecting apparatus, including:

a parameter obtaining unit configured to acquire parameter information of the synchronization data when the data of the specified file is synchronized, the parameter information being associated with the synchronization data;

a parameter confirmation unit, configured to determine whether there is a gap in each parameter information corresponding to the specified file that has been obtained;

The result confirming unit is configured to determine that the specified file has data loss when the parameter confirming unit determines that there is a gap.

Further, the parameter confirmation unit includes:

Inserting a subunit, configured to insert the obtained parameter information of the synchronization data into a parameter information linked list of the specified file, where the parameter information linked list includes parameter information corresponding to the specified file acquired earlier;

Merging the subunits, configured to merge adjacent parameter information in the parameter information linked list;

The confirmation subunit is configured to search for the parameter information of the vacant parameter information list in the merged parameter information list according to the preset condition.

Further, the confirmation sub-unit is configured to repeatedly search for the parameter information of the vacant parameter information list in the merged parameter information list according to the preset number of times threshold, and when the search result is the parameter information of the vacancy, the number of times of the search reaches the When the threshold value is used, it is determined that there is a gap in the parameter information corresponding to the acquired specified file.

Further, the device further includes:

The lost data determining unit is configured to determine data corresponding to the parameter information of the merged parameter information list that is hollowed out as data that is lost in the specified file.

Compared with the prior art, the embodiments of the present application include the following advantages:

In the embodiment of the present application, by using the parameter information of the specified file and the association relationship between the parameter information and the synchronization data of the specified file, it is only necessary to determine whether the obtained parameter information has a gap, thereby determining whether the specified file is in data synchronization. There is data loss. In this method, since the data synchronization of each specified file carries the parameter information, there is no problem of the sampling rate in the sampling check, so that the problem that the unsampled data cannot be detected when the data is lost is avoided, and The information is associated with the synchronization data of the specified file, and the parameter information can represent the data of the specified file at different synchronization times, so the detection of the target statistical record is incorrect when the data is repeatedly transmitted relative to the final consistency check. The problem. The method disclosed in the embodiment of the present application not only improves the detection accuracy of the lost data, but also has a simple and easy detection process and high detection efficiency.

DRAWINGS

1 is a flow chart showing the steps of an embodiment of a data detecting method of the present application;

2 is a flow chart showing the steps of a method for determining whether there is a gap in each parameter information corresponding to the specified file that has been obtained in the present application;

3 is a schematic diagram of a parameter information linked list in the present application;

4 is a structural block diagram of an embodiment of a data detecting apparatus of the present application;

Figure 5 is a block diagram showing the structure of a parameter confirming unit in the present application;

FIG. 6 is a structural block diagram of another embodiment of a data detecting apparatus of the present application.

detailed description

The above described objects, features and advantages of the present application will become more apparent and understood.

Referring to FIG. 1 , a flow chart of steps of an embodiment of a data detection method of the present application is shown, which may specifically include the following steps:

Step 101: When synchronizing the data of the specified file, acquiring parameter information of the synchronization data, where the parameter information is associated with the synchronization data.

In the embodiment of the present application, the device for detecting whether the data is lost (hereinafter referred to as the device) may be the target server itself when performing data synchronization, or a module set in the target server, or may be independent of the target. A device that is external to the server and that can interact with the target server.

When synchronizing the data of the specified file, for example, synchronizing the updated data of the log, the source server transmitting the data may send the data of the specified file together with the parameter information of the current data to the target server, and the device requests the target server. Obtaining parameter information of the synchronization data; or, the source server may separately send data of the specified file and parameter information of the current data to the target server and the device.

The parameter information may be newly added information, and the source server and the device agree that the source server carries the parameter information of the current data each time data synchronization is performed on the specified file. The parameter information is associated with the synchronization data, and the parameter information of the different data is different, and the parameter information may follow a certain rule, for example, the parameter information of the continuously transmitted synchronization data is continuous.

The parameter information may also be information that is included in an existing file, such as an offset and a data length. Each time the file is read, there will be two parameter information of offset (offset) and data length (length). When the source server reads the updated data of the file for data synchronization, the synchronous data is simultaneously transmitted and read. Offset and data length. The parameter information is associated with the synchronization data, and the offset of the next synchronization data is the sum of the offset of the synchronization data and the data length, namely:

Offset _next =offset _now +length _now

Among them, the length of each synchronization data may be the same or different.

The parameter information may also be information of other content, as long as the parameter information is associated with the synchronization data of the specified file, and may be used to distinguish the synchronization data of the specified file at each synchronization.

Step 102: Determine whether there is a gap in each parameter information corresponding to the obtained specified file.

After obtaining the parameter information in the previous step, the step may determine whether there is a gap in the parameter information according to the parameter information corresponding to all the obtained specified files. Specifically, it may be determined according to the law between the parameter information whether there is a vacancy or missing parameter information.

For example, if the parameter information of the synchronous data that is continuously sent in the previous step is continuous, in this step, it can be determined whether the parameter information of the obtained specified file is continuous, and if it is continuous, there is no vacancy. If it is not continuous, the parameter information is missing.

For another example, if the upper step uses the offset and the data length as the parameter information, the offset between the offsets of the two consecutive synchronizations may be offset according to the offset and the data length that have been obtained _next =offset _now +length _now determines the offset of the neighbor, and then finds whether there is the determined offset of the neighbor in the obtained parameter information, and so on, and can determine the parameter information corresponding to the obtained specified file. Is there a gap in the middle?

If parameter information of other content is used, since the parameter information is associated with the synchronization data of the specified file, the law between the different parameter information may be utilized to determine whether there is a gap in the parameter information of the obtained specified file.

If it is determined that there is a vacancy, step 103 is performed. If there is no vacancy, step 104 is performed.

In step 103, it is determined that there is data loss in the specified file.

Since the source server performs data synchronization according to the data update of the specified file, if the parameter information obtained by the device is vacant, the data corresponding to the vacated parameter information is lost, and the data synchronization has a problem.

Step 104: Determine that there is no data loss in the specified file.

In another embodiment of the present application, the process of determining whether the parameter information corresponding to the specified file is vacant, as shown in FIG. 2, may further include:

In step 201, the parameter information of the acquired synchronization data is inserted into the parameter information linked list of the specified file, and the parameter information linked list includes the parameter information corresponding to the specified file acquired earlier.

In order to facilitate the determination of whether there is a vacancy in the parameter information, in this embodiment, a parameter information linked list of the specified file may be established, where the parameter information linked list includes all the obtained parameter information corresponding to the specified file, that is, The parameter information of the synchronization data obtained during each data synchronization process of the file. The parameter information linked list can arrange parameter information according to the order of the size of the parameter information, etc., to facilitate searching.

When the device obtains the parameter information of the new synchronization data of the specified file in step 101, the parameter information is inserted into the corresponding parameter information linked list according to the arrangement rule.

Step 202: Combine adjacent parameter information in the parameter information linked list.

After inserting the parameter information list, the adjacent parameter information is merged. The adjacent parameter information is also parameter information of the synchronization data in the adjacent two data synchronizations. Specifically, the two parameter information may be adjacent according to the setting rule of the parameter information. For example, if the parameter information corresponding to the continuous synchronization data is continuous, the continuous parameter information is adjacent parameter information, which may be combined; The above offset _next =offset _now +length _now sets the parameter information, and based on the data length, it is estimated whether the two parameter information is adjacent, and then whether it can be merged.

Step 203: Search for the parameter information of the vacant parameter information in the merged parameter information list according to the preset condition.

In this step, the parameter information that cannot be merged may be the parameter information of the vacancy. When determining whether there is vacant parameter information, the condition may also be set, and if the search result satisfies the preset condition, it is determined whether there is a vacancy parameter. information.

For example, the threshold value of the number of times of the search is set in advance, and the parameter information of the vacant parameter information list is repeatedly searched according to the preset number of times threshold. When the search result is the parameter information of the vacancy and the number of times of the search reaches the threshold of the number of times, the judgment has been made. The parameter information corresponding to the obtained specified file is vacant.

In addition, after the parameter information of the vacancy is determined, the data corresponding to the parameter information of the merging parameter information list may be determined as the data lost in the data synchronization process of the specified file.

The following takes the parameter information as offset and length as an example for description. Each time the data is synchronized to the specified file, these two parameters are carried and saved in the target server. In the target server, the parameter information and the real synchronization data are saved, and the result state must be consistent, with success or failure. The means for detecting whether the data is lost may scan the parameters of the specified file saved by the target server to determine whether there is data loss.

The device establishes a parameter information linked list (TailHead) corresponding to the specified file, continuously reads parameter information of the new synchronous data from the target server, and inserts the parameter information into the parameter information linked list. For ease of understanding, it is assumed that the length of the parameter information obtained by each data synchronization is always 1, and the established parameter information list is as shown in FIG. 3, taking the d1 object of TailHead as an example, and the tail is 1 to indicate that its offset is 1, length. A value of 1, head is 5 indicates that the offset is 5 and the length is 1. D1 indicates that the synchronization data between 1 and 5 has been detected. As shown in FIG. 3, the data in the figure indicates that the synchronization data in 1 to 5, 8 to 10, 12, and m to n already exist, and is not lost. .

When the apparatus performs the foregoing step 101, when the parameter information of the new synchronization data of the specified file is read from the target server, for example, the offset of the read sync data is 11 and the length is 1, and the foregoing step 201 is executed, and the read is performed. The parameter information is inserted into the linked list TailHead, then the tail value of d2 is updated to 11 according to the parameter information offset of 11 and length being 1. Then, the foregoing step 202 is performed to merge adjacent parameter information in the linked list. For example, the tail of d2 is 11 and the offset is 11, the length is 1, and the head of d3 is 12, that is, the offset is 12, and the length is 1, then 1. It can be determined that d2 is adjacent to d3, and d2 and d3 are combined, wherein head is 8 and tail is 13. Then, the foregoing step 203 is performed to detect whether there is a gap in the parameter information. Taking d1 and d2 as an example, the offset between them is 6, 7 indicating that the data corresponding to the two parameter information does not exist. If the two parameters are judged to be absent after multiple times (the number of thresholds can be adjusted), then the data corresponding to offsets 6 and 7 can be considered to be lost.

It should be noted that, for the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the embodiments of the present application are not limited by the described action sequence, because In accordance with embodiments of the present application, certain steps may be performed in other sequences or concurrently. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required in the embodiments of the present application.

Referring to FIG. 4, a structural block diagram of an embodiment of a data detecting apparatus of the present application is shown, which may specifically include the following units:

The parameter obtaining unit 401 is configured to acquire parameter information of the synchronization data when the data of the specified file is synchronized, and the parameter information is associated with the synchronization data.

The parameter confirmation unit 402 is configured to determine whether each parameter information corresponding to the specified file that has been acquired has a gap.

The result confirming unit 403 is configured to determine that the specified file has data loss when the parameter confirming unit determines that there is a gap.

In the embodiment of the present application, by using the parameter information of the specified file and the association relationship between the parameter information and the synchronization data of the specified file, the device only needs to determine whether the obtained parameter information has a gap, thereby determining that the specified file is in the Whether there is data loss when data is synchronized. The device not only improves the detection accuracy of the lost data, but also has a simple and easy detection process and high detection efficiency.

In another example, when the parameter confirmation unit determines that there is no vacancy, the result confirmation unit 403 may determine that there is no data loss in the specified file.

In another embodiment, as shown in FIG. 5, the parameter confirmation unit 402 may further include:

The insertion sub-unit 501 is configured to insert the acquired parameter information of the synchronization data into the parameter information linked list of the specified file, where the parameter information linked list includes parameter information corresponding to the specified file acquired earlier.

The merging subunit 502 is configured to merge adjacent parameter information in the parameter information linked list.

The confirmation subunit 503 is configured to search for the parameter information of the vacancy in the merged parameter information list according to the preset condition.

The confirmation sub-unit 503 may be specifically configured to repeatedly search for the parameter information of the vacant parameter information list in the merged parameter information list according to a preset number of times threshold, and when the search result is the parameter information of the vacancy, the number of times of finding reaches When the threshold value is described, it is determined that there is a gap in the parameter information of the specified file that has been acquired.

In another embodiment, as shown in FIG. 6, the device may further include:

The lost data determining unit 601 is configured to determine data corresponding to the parameter information of the merged parameter information list that is hollowed out as data that is lost in the specified file.

The parameter information of the above synchronization data may include an offset and a data length, and the offset of the next synchronization data is the sum of the offset of the synchronization data and the data length.

The embodiment of the present application also discloses an electronic device, including a memory and a processor.

The processor and the memory are connected to each other through a bus; the bus may be an ISA bus, a PCI bus, or an EISA bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

Wherein, the memory is used to store a program, and specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include a high speed RAM memory and may also include a non-volatile memory such as at least one disk memory.

The processor is used to read the program code in the memory and perform the following steps:

If yes, it is determined that there is data loss in the specified file.

For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment.

The various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on differences from other embodiments, and the same similar parts between the various embodiments can be referred to each other.

Those skilled in the art should appreciate that embodiments of the embodiments of the present application may be provided as a method, apparatus, or calculation. Machine program products. Therefore, the embodiments of the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, embodiments of the present application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

In a typical configuration, the computer device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory. The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium. Computer readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic tape cartridges, magnetic tape storage or other magnetic storage devices or any other non-transportable media can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-persistent computer readable media, such as modulated data signals and carrier waves.

Embodiments of the present application are described with reference to flowcharts and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing terminal device to produce a machine such that instructions are executed by a processor of a computer or other programmable data processing terminal device Means are provided for implementing the functions specified in one or more of the flow or in one or more blocks of the flow chart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The instruction device implements the functions specified in one or more blocks of the flowchart or in a flow or block of the flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device such that a series of operational steps are performed on the computer or other programmable terminal device to produce computer-implemented processing, such that the computer or other programmable terminal device The instructions executed on the implementation are used to implement a process or multiple processes in the flowchart And/or block diagram of the steps of a function specified in a box or blocks.

While a preferred embodiment of the embodiments of the present application has been described, those skilled in the art can make further changes and modifications to the embodiments once they are aware of the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the modifications and the modifications

Finally, it should also be noted that in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities. There is any such actual relationship or order between operations. Furthermore, the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, or terminal device that includes a plurality of elements includes not only those elements but also Other elements that are included, or include elements inherent to such a process, method, article, or terminal device. An element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that comprises the element, without further limitation.

The data detection method and a data detection device provided by the present application are described in detail above. The principles and implementation manners of the present application are described in the specific examples. The description of the above embodiments is only for helping. The method of the present application and its core idea are understood; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation manner and the scope of application. It should be understood that the limitations of the application.

Claims

A data detection method, comprising:

Obtaining parameter information of the synchronization data when the data of the specified file is synchronized, the parameter information being associated with the synchronization data;

Determining whether there is a gap in each parameter information corresponding to the specified file that has been obtained;

If yes, it is determined that there is data loss in the specified file.
The method according to claim 1, wherein the determining whether the obtained parameter information corresponding to the specified file has a gap includes:

The parameter information of the acquired synchronization data is inserted into the parameter information linked list of the specified file, where the parameter information linked list includes parameter information corresponding to the specified file acquired earlier;

Merging adjacent parameter information in the parameter information linked list;

The parameter information of the vacant parameter information in the merged parameter information list is found according to the preset condition.
The method according to claim 2, wherein the parameter information of the vacant parameter list in the merged parameter information list is searched according to a preset condition, including:

Determining, by using a predetermined number of thresholds, whether there is a parameter information of the vacancy in the merged parameter information list. When the search result is the parameter information of the vacancy and the number of times of searching reaches the threshold number, the determined location is determined. There is a gap in the parameter information corresponding to the specified file.
The method of claim 2, wherein the method further comprises:

Determining, by the merged data corresponding to the parameter information of the parameter information list, the data of the specified file is lost.
The method according to any one of claims 1 to 4, wherein the parameter information of the synchronization data includes an offset and a data length, and the offset of the next synchronization data is a bias of the synchronization data. The sum of the shift and the length of the data.
A data detecting device, comprising:

a parameter obtaining unit configured to acquire parameter information of the synchronization data when the data of the specified file is synchronized, the parameter information being associated with the synchronization data;

a parameter confirmation unit, configured to determine whether there is a gap in each parameter information corresponding to the specified file that has been obtained;

The result confirming unit is configured to determine that the specified file has data loss when the parameter confirming unit determines that there is a gap.
The device according to claim 6, wherein the parameter confirmation unit comprises:

Inserting a subunit, configured to insert the obtained parameter information of the synchronization data into a parameter information linked list of the specified file, where the parameter information linked list includes parameter information corresponding to the specified file acquired earlier;

Merging the subunits, configured to merge adjacent parameter information in the parameter information linked list;

The confirmation subunit is configured to search for the parameter information of the vacant parameter information list in the merged parameter information list according to the preset condition.
The device of claim 7 wherein:

The acknowledgment sub-unit is configured to repeatedly search for the parameter information of the vacant parameter information list in the merged parameter information list according to the preset number of times thresholds, and when the search result is the vacancy parameter information, and the number of times of the search reaches the threshold number of times When it is determined, there is a gap in the parameter information corresponding to the obtained specified file.
The device according to claim 7, wherein the device further comprises:

The lost data determining unit is configured to determine data corresponding to the parameter information of the merged parameter information list that is hollowed out as data that is lost in the specified file.
The apparatus according to any one of claims 6 to 9, wherein the parameter information of the synchronization data includes an offset amount and a data length, and the offset of the next synchronization data is a bias of the synchronization data. The sum of the shift and the length of the data.