WO2024020898A1 - Data error detection method, apparatus, electronic device, and storage medium - Google Patents

Abstract

Disclosed in the embodiments of the present invention are a data error detection method, an apparatus, an electronic device, and a storage medium. The method comprises: determining data types contained in template data; on the basis of statistical analysis of values of the data types in the template data, determining statistical characteristics of the data types; on the basis of the data types and the statistical characteristics of the data types, determining a structure file; receiving test data; comparing the test data with the structure file; and, on the basis of the comparison result, determining whether the test data has had an error. The embodiments of the present invention determine, on the basis of the data types in the template data and the statistical characteristics of the data types, the structure file serving as a data comparison target; and by means of comparing the test data with the structure file, the test data having had an error can be quickly detected, thus improving the data error detection efficiency and the data correction efficiency.

Description

Data error detection method, device, electronic equipment and storage medium Technical field

The present invention relates to the field of data processing technology, in particular to data error detection methods, devices, electronic equipment and storage media.

Background technique

The impact of the Internet on data is huge. Moreover, industrial development is increasingly driven by data-based technologies. In order to make better use of data, some data integration solutions or tools, middleware, low-code tools and visualization software are gradually solidified.

However, in the face of information silos and many scattered data, even if some data integration solutions can be used to achieve system or data integration, data quality problems may still be encountered. Data quality issues may run through all service levels of the entire digital construction. For example, data quality problems may be encountered in many scenarios such as system integration, application transplantation, on-site data collection equipment failure, format conversion caused by dependency library updates, expansion and development of new functions, etc.

Currently, how to quickly detect erroneous data is a technical problem to be solved.

Contents of the invention

The embodiments of the present invention provide data error detection methods, devices, electronic equipment and storage media.

A data error detection method, the method includes:

Determine the data type contained in the template data;

Determine the statistical characteristics of the data type based on counting the values of the data type in the template data;

Determine the structure file based on the data type and the statistical characteristics of the data type;

receive test data;

Compare the test data with the structure file;

Based on the result of the comparison, it is determined whether the test data is in error.

It can be seen that the embodiment of the present invention determines the structural file as the data comparison target based on the data type in the template data and the statistical characteristics of the data type. By comparing the test data with the structural file, it can quickly determine whether the test data is wrong and improve Improve error detection efficiency.

In an exemplary embodiment, determining whether the test data is in error based on a result of the comparison includes at least one of the following:

When the data type contained in the test data does not match the data type in the structure file, it is determined that the test data is in error;

When the value of the data type in the test data does not comply with the statistical characteristics in the structure file, it is determined that the test data is in error;

When the value of the data type in the test data does not conform to the statistical characteristics in the structure file and the data type contained in the test data does not match the data type in the structure file, it is determined that the The above test data is wrong;

The test is determined when the value of the data type in the test data conforms to the statistical characteristics in the structure file and the data type contained in the test data matches the data type in the structure file. The data is normal.

Therefore, the embodiment of the present invention can detect errors in a variety of ways, thereby improving applicability.

In an exemplary embodiment, it also includes:

When the test data is erroneous, performing a correction operation on the test data based on the structure file;

In the log file, data error events and the corrective actions are recorded.

It can be seen that the embodiment of the present invention can correct erroneous test data and improve data quality. Moreover, by recording data error events and correction operations in log files, data quality logging is achieved, making it easier for subsequent processors to understand the data status in a timely manner when taking over, integrating or migrating data.

In an exemplary embodiment, performing a correction operation on the test data based on the structure file includes at least one of the following:

Based on the statistical characteristics of the data type contained in the structure file, correct the value of the corresponding data type in the test data;

Based on the data type contained in the structure file, the data type in the test data is corrected.

Therefore, the embodiment of the present invention can perform various types of accurate corrections on erroneous data, thereby improving data quality.

In an exemplary implementation, the data types contained in the determined) template data include:

Determine the first data type in the template data and the object in the template data;

Determine a second data type contained by the object;

The first data type and the second data type are determined as data types included in the template data.

It can be seen that the embodiment of the present invention can extract the data type of the template data and the data type of the object in the template data, thereby improving the richness of the data type.

In an exemplary embodiment, it also includes:

Display the structure file on the human-computer interaction interface;

Receive structural file adjustment instructions triggered via the human-computer interaction interface;

The data type in the structure file and/or the statistical characteristics in the structure file are adjusted based on the structure file adjustment instruction.

Therefore, the embodiment of the present invention can adjust the structure file based on user-triggered operations, enriching the flexibility of data processing.

In an exemplary embodiment, the statistical characteristics include at least one of the following:

The maximum length of the data type; the minimum length of the data type; the maximum value of the data type; the minimum value of the data type; the average value of the data type; the maximum time interval of the data type; The minimum time interval for the above data type; the average time interval for the above data type.

It can be seen that the data types of the embodiments of the present invention are diverse and suitable for various application environments.

In an exemplary embodiment, the template data is contained in an instance field of an application programming interface design file, or in a real-time file obtained via real-time access to the application programming interface.

Therefore, the data sources of template data are diverse and suitable for various application environments.

In an exemplary implementation, before determining the data type contained in the template data, the method further includes:

Parse the template data from input data having multiple protocol encapsulation formats;

Convert the template data into a predetermined data exchange format.

Therefore, by converting the template data into a data exchange format, data workers can process the template data without a deep understanding of the communication interaction process.

A data processing device, the device includes:

The first determination module is configured to determine the data type contained in the template data;

The second determination module is configured to determine the statistical characteristics of the data type based on statistics of the values of the data type in the template data;

The third determination module is configured to determine the structure file based on the data type and the statistical characteristics of the data type;

The receiving module is configured to receive test data;

a comparison module configured to compare the test data with the structure file;

A fourth determination module is configured to determine whether the test data is in error based on the result of the comparison.

It can be seen that the embodiment of the present invention determines the structural file as the data comparison target based on the data type in the template data and the statistical characteristics of the data type. By comparing the test data with the structural file, it can quickly determine whether the test data is wrong and improve Improve error detection efficiency.

In an exemplary implementation, the fourth determining module is configured to perform at least one of the following:

When the data type contained in the test data does not match the data type in the structure file, it is determined that the test data is in error;

When the value of the data type in the test data does not comply with the statistical characteristics in the structure file, it is determined that the test data is in error;

When the value of the data type in the test data does not conform to the statistical characteristics in the structure file and the data type contained in the test data does not match the data type in the structure file, it is determined that the The above test data is wrong;

The test is determined when the value of the data type in the test data conforms to the statistical characteristics in the structure file and the data type contained in the test data matches the data type in the structure file. The data is normal.

Therefore, the embodiment of the present invention can detect errors in a variety of ways, thereby improving applicability.

In an exemplary embodiment, it also includes:

A correction module configured to perform a correction operation on the test data based on the structure file when the test data has an error; record a data error event and the correction operation in a log file.

It can be seen that the embodiment of the present invention can correct erroneous test data and improve data quality. Moreover, by recording data error events and correction operations in log files, document logging of data quality is achieved, allowing subsequent processors to understand the data status in a timely manner when taking over, integrating or migrating data.

In an exemplary implementation, the correction module is configured to perform at least one of the following:

Based on the statistical characteristics of the data type contained in the structure file, correct the value of the corresponding data type in the test data;

Based on the data type contained in the structure file, the data type in the test data is corrected.

Therefore, the embodiment of the present invention can perform various types of accurate corrections on erroneous data, thereby improving data quality.

In an exemplary implementation, the first determination module is configured to determine a first data type in the template data and an object in the template data; determine a second data type contained in the object; and convert the The first data type and the second data type are determined to be the data types included in the template data.

It can be seen that the embodiment of the present invention can extract the data type of the template data and the data type of the object in the template data, thereby improving the richness of the data type.

In an exemplary embodiment, it also includes:

an adjustment module configured to display the structure file on a human-computer interaction interface; receive a structure file adjustment instruction triggered via the human-computer interaction interface; and adjust the data type in the structure file based on the structure file adjustment instruction, and/or statistical features in said structure file.

Therefore, the embodiment of the present invention can adjust the structure file based on user-triggered operations, enriching the flexibility of data processing.

An electronic device including:

processor;

memory for storing executable instructions for the processor;

The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the data error detection method as described in any one of the above.

A computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the data error detection method as described in any one of the above items is implemented.

A computer program product includes a computer program that implements the data error detection method as described in any of the above items when executed by a processor.

Description of drawings

Preferred embodiments of the present invention will be described in detail below to make the above and other features and advantages of the present invention more apparent to those skilled in the art with reference to the accompanying drawings, in which:

FIG. 1 is an exemplary flow chart of a data error detection method according to an embodiment of the present invention.

Figure 2 is a flow chart of a method for determining a structure file according to an embodiment of the present invention.

3 is a schematic diagram of an exemplary process of determining a structured file using a real-time file according to an embodiment of the present invention.

Figure 4 is an exemplary schematic diagram of a data correction process according to an embodiment of the present invention.

FIG. 5 is an exemplary schematic diagram of a process of generating a structure file and performing data correction according to an embodiment of the present invention.

FIG. 6 is an exemplary structural diagram of a data error detection device according to an embodiment of the present invention.

FIG. 7 is an exemplary structural diagram of a data error detection device with a processor-memory architecture according to an embodiment of the present invention.

Among them, the reference signs are as follows:

label	meaning
100	Data error detection methods
101～106	step
201～210	step
40	template data
41	path object
42	Server object
411	First endpoint sub-object
412	Second endpoint sub-object
413	Third endpoint sub-object
50	path item object
51	Get operation
52	Put operation
53	Submit operation
54	Delete operation
60	Operation object
61	Describe object
62	Operation object ID
30	variable table object
31	variable object
311	string
312	numerical value
313	queue

314	time date
411	Statistical characteristics of time and date
412	Statistical characteristics of numerical values
413	Statistical characteristics of the queue
414	Statistical characteristics of time and date
401～409	step
80	real time file
81	Instance fields of API design files
90	data processing device
82	protocol adapter
83	Structure file generator
831	structure parser
832	feature parser
84	writer
85	display processing
91	Adjustment processing
92	Structure file processing layer
86	Test Data
87	protocol adapter
88	data processor
881	Error detector
882	Correction
89	Alarm
600	data processing device

601	First determination module
602	second determination module
603	The third determination module
604	receiving module
605	compare module
606	The fourth determination module
607	Calibration module
608	Adjustment module
700	data processing device
701	processor
702	memory

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the following examples are given to further describe the present invention in detail.

For the sake of simplicity and intuitiveness in description, the solution of the present invention is explained below by describing several representative embodiments. A large number of details in the embodiments are only used to help understand the solution of the present invention. However, it is obvious that the technical solution of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the solutions of the present invention, some embodiments are not described in detail, but only give a framework. Hereinafter, "including" means "including but not limited to", and "based on..." means "at least based on..., but not limited to only based on...". Due to Chinese language habits, when the number of a component is not specified below, it means that the component can be one or more, or it can be understood as at least one.

The applicant analyzed the current status of data governance and found that there are at least the following problems:

(1). It is difficult to find data quality problems. For example, in Internet of Things projects, it is difficult to detect abnormal situations such as sensor drop, data drift, edge communication network interruption, and data processing algorithm formulas that cannot adapt to changing data. Moreover, changes in the deployment environment and upgrades of dependent libraries can also lead to incorrect data or incorrect date formats. These data quality problems that usually lurk but subsequently erupt require data workers who are familiar with the entire system to find solutions.

(2) It takes a lot of time and effort to understand the communication interaction data. Data workers need to spend a lot of time and effort to understand the interactive data of each developed application or system, such as what communication interfaces exist and the data and business requirements contained in the interfaces (e.g., data format, meaning, update cycle, etc. )

(3). There is currently no documentation or logging on data quality issues, so there is no easy way to verify data quality issues. Integration testing for data quality is addressed on a case-by-case basis when taking over, integrating, or porting an application or system due to a lack of documentation and event logging regarding data quality.

In the embodiment of the present invention, a unified and scalable solution is provided for flexibly implementing data quality management (including error detection and correction). The embodiments of the present invention can be integrated into systems, clouds, platform tools or plug-ins, etc., which can make data quality management easier and more efficient, and can especially effectively support industrial data services and applications.

FIG. 1 is an exemplary flow chart of a data error detection method according to an embodiment of the present invention. As shown in Figure 1, the method 100 includes:

Step 101: Determine the data type contained in the template data.

Here, the template data is reference data used to perform data error detection. For example, template data can be implemented as data contained in instance fields of an application programming interface (API) design file. Template data can also be implemented as data contained in a live file, where the live file is obtained via the live access API interface.

In system construction, API design files are common. API design files usually contain various instance fields. The data in these instance fields can be used as reference data in the data error detection process, that is, template data. API design files are usually static files. API design files can be obtained from various data sources (for example, various development documents). For example, the format of the API design file can include JSON, YML, etc., but is not limited to these formats. API design files can be exported from some API design and testing tools such as Postman or OpenAPI, etc.

Users can also access the API interface in real time to obtain real-time files. The data in the real-time file can also be used as reference data in the data error detection process, that is, template data. Specifically, the system protocols for accessing the API interface may include: Hyper Text Transfer Protocol (HTTP), Hyper Text Transfer Protocol over Secure Socket Layer (HTTPS) based on Secure Socket Layer, and messages. Queuing Telemetry Transport (Message Queuing Telemetry Transport, MQTT) protocol, etc., but are not limited to these protocols and support protocol extensions.

Preferably, the template data, whether derived from static API design files or real-time files, can be converted into a unified format, such as JSON format, to facilitate subsequent unified processing. In one implementation, the template data is obtained from the instance field of the API design file, and the template data in the real-time file is obtained through the real-time access API interface, and the two paths of template data are merged into the overall template data, and then the merged The final template data is converted into JSON format to facilitate subsequent unified processing.

After the template data is obtained, the template data is parsed to determine the data type contained in the template data. In one embodiment, step 101 includes: determining the first data type in the template data and the object in the template data; determining the second data type contained in the object; determining the first data type and the second data type. The data type contained in the template data. When the object also contains sub-objects, the data type contained in the sub-object can be further determined, and the data type contained in the sub-object can be determined into the data type of the template data.

For example, data types include: numerical type (such as integer type or floating point type); character type; string type; Boolean type; queue (array) type; time and date type, etc.

The above exemplarily describes typical examples of data types. Those skilled in the art can realize that this description is only exemplary and is not used to limit the protection scope of the embodiments of the present invention.

Step 102: Determine the statistical characteristics of the data type based on counting the values of the data type in the template data.

Here, the values of each data type in the template data are parsed to determine the statistical characteristics of the data type. For example, the statistical characteristics may include at least one of the following: the maximum length of the data type; the minimum length of the data type; the maximum value of the data type; the minimum value of the data type; the average value of the data type; the maximum time interval of the data type; The minimum time interval for a type; the average time interval for a data type, etc.

An exemplary process for determining statistical characteristics of template data is described below. Assume that the template data contains a time series object, and the time series object contains a queue type. The queue type contains multiple objects with their own data types, namely timestamp (timestamp), vibration extreme value (vibmax), vibration mean ( vibmean), vibration effective value (vibrms), vibration skewness (vibskewness), vibration kurtosis (vibkurtosis), temperature (temperature) and humidity (humidity). Among them, the data type of the timestamp is time and date type (data time); the data type of vibration extreme value, vibration mean value, vibration effective value, vibration skewness, vibration kurtosis, temperature and humidity is numerical type (number). The template data also contains values for each type in the queue. The statistical characteristics of each data type can be statistically calculated based on the value of each data type.

For example, the values of timestamps (data type specifically data time) representing multiple time points when multiple measurement experiments are performed are: 2020-07-16 01:00:04, 2020-07-16 01:00:10 , 2020-07-16 01:00:16, 2020-07-16 01:00:22 and 2020-07-16 01:00:28. By analyzing these values, it can be found that the time stamp data interval of each measurement test is 6 seconds. Therefore, the statistical characteristics of the timestamp can be obtained as follows: the maximum time interval of the timestamp data is 6 seconds; the minimum time interval of the timestamp data is 6 seconds. ;The average time interval of timestamp data is 6 seconds.

For another example, the multiple vibration extreme values (data type is number) representing multiple measurement tests are: 72.0, 80.0, 82.0, 78.0 and 79.0 respectively. By analyzing these values, the statistical characteristics can be obtained as follows: the maximum value of the vibration extreme value is 82 seconds; the minimum value of the vibration extreme value is 72 seconds; and the average value of the vibration extreme value is 78.2.

Similarly, the statistical characteristics of each data type in the template data can be calculated. Among them, the more statistical features, usually the higher the error correction efficiency for this data type.

Step 103: Determine the structure file based on the data type and the statistical characteristics of the data type.

Here, the correspondence between each data type and its statistical characteristics is established. A file containing all correspondences between data types and their statistical characteristics is a structure file.

Following the above example, an exemplary example of a structure file is described. For example, a time series contains a queue, and the queue contains multiple objects with their own data types. The data types in the queue include the time and date type of the timestamp, as well as vibration extreme value, vibration mean value, vibration effective value, vibration Numeric types for skewness, vibration kurtosis, temperature, and humidity. Moreover, each data type in the queue has its own statistical characteristics. Each data type in the queue is stored in association with its respective statistical characteristics to obtain a structure file.

The above are typical examples of exemplary structure files. Those skilled in the art can realize that this description is only exemplary and is not used to limit the protection scope of the embodiments of the present invention.

After the structure file is determined in steps 101 to 103, the structure file can be used to perform data error detection and correction processes.

Step 104: Receive test data.

Here, the test data is the object on which data error checking is performed. For example, when the template data is implemented as data contained in the instance field of the API design file, the test data can be the data output by the API interface corresponding to the API design file within the specified test time. For another example, when the template data is implemented as data contained in a real-time file, the test data can be data output during the test time via an API interface that provides the real-time file.

Step 105: Compare the test data with the structure file.

Here, the test data is compared to the structure file, including doing at least one of the following:

(1): Compare the types in the test data with the types in the structure file;

(2): Compare the values of the types in the test data to see whether they conform to the statistical characteristics of the corresponding types in the structure file.

Step 106: Based on the comparison result, determine whether the test data is in error.

In one embodiment, step 106 includes the following scenarios:

(1) When the data type contained in the test data does not match the data type in the structure file, it is determined that the test data is wrong.

For example, if the test data contains data type A, data type B, and data type C, but the data type in the structure file only contains data type A and data type B, then it is deemed that the data types of the test data do not match, and the test data is deemed to be mismatched. Something went wrong.

(2) When the value of the data type in the test data does not conform to the statistical characteristics in the structure file, it is determined that the test data is wrong.

For example, if the value of data type A contained in the test data is 100, but the statistical characteristics of data type A in the structure file indicate that the maximum value is 60, then the test data is deemed to be wrong.

(3) When the value of the data type in the test data does not conform to the statistical characteristics in the structure file and the data type contained in the test data does not match the data type in the structure file, it is determined that the test data is wrong.

For example, the test data contains data type A, data type B, and data type C, and the value of data type A in the test data is 100. If the data types in the structure file only contain data type A and data type B, and the statistical characteristic indication of data type A in the structure file has a maximum value of 60, it is deemed that the test data is wrong.

(4) When the value of the data type in the test data conforms to the statistical characteristics in the structure file and the data type contained in the test data matches the data type in the structure file, the test data is determined to be normal.

For example, the test data contains data type A, data type B and data type C, and the value of data type A in the test data is 60. If the data types in the structure file include data type A, data type B, and data type C, and the statistical characteristic indicator of data type A in the structure file has a maximum value of 100, then the test data is considered normal.

The above exemplarily describes a typical example of determining whether the test data is erroneous based on the comparison results. Those skilled in the art can realize that this description is only exemplary and is not used to limit the protection scope of the embodiments of the present invention.

In one embodiment, before step 101, the method further includes: parsing template data from input data having multiple protocol encapsulation formats; and converting the template data into a predetermined data exchange format. For example, the predetermined data exchange format can be implemented as JSON format, and the protocols can include: Modbus communication protocol, RS-323 protocol, RS-485 protocol and HART protocol, etc. Therefore, by converting the template data into a predetermined data exchange format, data workers can process the template data without a deep understanding of the communication interaction process.

In one embodiment, the method further includes: when the test data has an error, performing a correction operation on the test data based on the structure file; recording the data error event and the correction operation in the log file. It can be seen that the embodiment of the present invention can correct erroneous test data and improve data quality. Moreover, by recording data error events and correction operations, document logging of data quality is achieved, allowing subsequent processors to understand the data status in a timely manner when taking over, integrating or migrating data.

In one embodiment, performing a correction operation on the test data based on the structure file includes at least one of the following: correcting the value of the corresponding data type in the test data based on the statistical characteristics of the data type contained in the structure file; based on the statistical characteristics of the data type contained in the structure file; Data type, correct the data type in the test data. For example, when the time format in the test data does not match the time format in the structure file, the time format in the test data is corrected based on the time format in the structure file. For another example, when the numerical value in the test data is wrong (for example, the parameter value that should be fixed is wrong), the numerical value in the test data is corrected based on the correct value in the structure file. Therefore, the embodiment of the present invention can perform various types of accurate corrections on erroneous data, thereby improving data quality.

Figure 2 is a flow chart of a method for determining a structure file according to an embodiment of the present invention.

As shown in Figure 2, the method includes:

Step 201: Obtain input data, where the input data may have multiple protocol encapsulation formats.

Step 202: Determine whether the input data comes from the real-time file obtained through the real-time access API interface. If so (corresponding to the "Y" branch), execute step 203 and subsequent steps; otherwise (corresponding to the "N" branch), execute Step 210 and subsequent steps.

Step 203: Start protocol adaptation processing (Protocol adaptation processing) on the input data from the real-time file.

Step 204: In the protocol adaptation process, template data is parsed from input data having multiple protocol encapsulation formats.

Step 205: Convert the template data into JSON format.

Step 206: Extract the data type contained in the template data in JSON format.

Step 207: Count the values of each data type in the template data to determine the statistical characteristics of each data type.

Step 208: Write the data type and the statistical characteristics of the data type into the structure file in an associated manner.

Step 209: Obtain the structure file used as the data comparison target and used to perform data error correction and data correction, and end this process.

Step 210: Start executing protocol adaptation processing on the input data from the non-real-time file, parse the template data, and jump to step 205.

The above describes an exemplary process of determining the structure file. 3 is a schematic diagram of an exemplary process of determining a structured file using a real-time file according to an embodiment of the present invention.

In FIG. 3 , the template data 40 obtained based on the real-time file includes: a path object 41 and a server object 42 . Exemplarily, the template data 40 may also include an API name (api) object or a path (paths) object, and so on.

For example, by parsing the path object 41, it is found that the path object 41 contains multiple sub-objects, namely the first endpoint sub-object 411, the second endpoint sub-object 412 and the third endpoint sub-object 413, etc., and each The child objects of each endpoint further point to the path item object 50. The path item object 50 includes sub-objects such as a get operation (get) 51, a put operation (put) 52, a submit operation (post) 53, and a delete operation (delete) 54. Each sub-object of the path item object 50 also points to the operation object 60 . The operation object 60 includes a description object 61 and an operation object identification 62, and so on. Similarly, the structures of the remaining objects in the template data 40 except the path object 41 are analyzed. Based on the object structure topology of the template data, a similar object structure topology is established in the structure file.

The data type contained in each object in the template data 40 is determined. For example, variable table object 30 is a child object of server object 42. The variable table object 30 contains a variable object 31 . The variable object 31 contains multiple data types, namely string (string) 311, numerical value (number) 312, queue (array) 313 and time and date 314. Based on the statistics of the values of these data types, statistical characteristics of each data type can be obtained. For example, for the time and date type 314, the statistical characteristics 414 may include the maximum time interval (maxInterval) and the minimum time interval (minInterval), and so on. For queue 313, statistical characteristics 413 may include maximum queue length (maxNum), minimum queue length (minNum), average queue length (aveNum), and so on. For the character string 311, the statistical features 411 may include the maximum character string length (maxLen), the minimum character string length (minLen), whether it overlaps (OverLap), and so on. For the value 312, the statistical characteristics 412 may include the length of the value (numLen), whether it is zero (isZero), whether it is empty (isNul), and so on.

Figure 4 is an exemplary schematic diagram of a data correction process according to an embodiment of the present invention. As shown in Figure 4, the data correction process includes:

Step 401: Obtain the input data within the test time through the API interface.

Step 402: Start protocol adaptation processing on the input data.

Step 403: In the protocol adaptation process, parse the test data from the test data with multiple protocol encapsulation formats.

Step 404: Convert the test data to JSON format.

Step 405: Compare the test data in JSON format with the structure file.

Step 406: Based on the comparison result, determine whether the test data is wrong. If there is an error (corresponding to the "Y branch"), execute step 407 and subsequent steps; otherwise (corresponding to the "N" branch), execute step 409.

Step 407: Correct the test data based on the structure file.

Step 408: Record error events and corrective operations.

Step 409: Output test data.

FIG. 5 is an exemplary schematic diagram of a process of generating a structure file and performing data correction according to an embodiment of the present invention.

As shown in FIG. 5 , the real-time file 80 received via the API interface and the instance field 81 of the API design file are provided to the data processing device 90 as input data. The protocol adapter 82 in the data processing device 90 performs protocol adaptation processing on the input data. The data processing device 90 includes a structure file generator 83 . The structure parser 831 in the structure file generator 83 determines the data type in the input data. The feature parser 832 in the structure file generator 83 determines the statistical features of the data type. The writer 84 in the data processing device 90 associates the data type determined by the structure parser 831 with the statistical characteristics determined by the feature parser 832 and writes them into the structure file. The structure document processing layer 92 includes presentation processing 85 and adjustment processing 91 . The structure file is displayed in the display process 85 . In the adjustment process 91, the structure file is adjusted based on the user-triggered operation.

At this point, the generation and adjustment processing of the structure file is completed. Structure files can be used to perform data error detection and data correction.

Test data 86 is provided to data processing device 90 . The protocol adapter 87 in the data processing device 90 performs protocol adaptation processing on the test data 86 . Data processing means 88 includes a data processor 88 . The data processor 88 includes an error detector 881 and a corrector 882 . The error detector 881 compares the test data 86 with the structure file obtained from the structure file processing layer 92 to check whether the test data 86 has errors. Corrector 882 performs correction on erroneous test data 86 . When an error occurs in the test data 86, the alarm 89 performs alarm processing, and stores and records data error events and correction operations.

FIG. 6 is an exemplary structural diagram of a data error detection device according to an embodiment of the present invention. As shown in Figure 6, the data processing device 600 includes:

The first determination module 601 is configured to determine the data type contained in the template data;

The second determination module 602 is configured to determine the statistical characteristics of the data type based on the value of the data type in the statistical template data;

The third determination module 603 is configured to determine the structure file based on the data type and the statistical characteristics of the data type;

The receiving module 604 is configured to receive test data;

The comparison module 605 is configured to compare the test data with the structure file;

The fourth determination module 606 is configured to determine whether the test data is in error based on the comparison result.

In an exemplary embodiment, the fourth determination module 606 is configured to perform at least one of the following: when the data type contained in the test data does not match the data type in the structure file, determine that the test data has an error; when the test data contains When the value of the data type in the test data does not conform to the statistical characteristics in the structure file, it is determined that the test data is wrong; when the value of the data type in the test data does not conform to the statistical characteristics in the structure file and the data type contained in the test data is consistent with the data type in the structure file. When the data type does not match, it is determined that the test data is wrong; when the value of the data type in the test data conforms to the statistical characteristics in the structure file and the data type contained in the test data matches the data type in the structure file, it is determined that the test data is normal ,etc.

In an exemplary embodiment, a correction module 607 is also included, configured to perform correction operations on the test data based on the structure file when the test data is erroneous; record data error events and correction operations in the log file.

In an exemplary embodiment, the correction module 607 is configured to perform at least one of the following: correcting the values of the corresponding data types in the test data based on the statistical characteristics of the data types contained in the structure files; based on the data contained in the structure files Type, correct the data type in the test data.

In an exemplary implementation, the first determination module 601 is configured to determine the object in the template data; determine the data type contained in the object.

In an exemplary embodiment, an adjustment module 608 is also included, configured to display the structure file on the human-computer interaction interface; receive the structure file adjustment instruction triggered via the human-computer interaction interface; and adjust the data in the structure file based on the structure file adjustment instruction. type, and/or statistical characteristics in the structure file.

Embodiments of the present invention save effort and time to resolve data quality issues and automate data governance with ease. Use data features to quickly abstract standard patterns, reduce the pressure of data quality inspection, effectively record data error conditions and correct operation records, and ensure data quality and efficient team collaboration. Moreover, by automatically generating structure files, data governance can be flexibly implemented, and the module architecture can be easily extended and integrated to facilitate integration with existing products, systems or services. Embodiments of the present invention can implement and expand digital systems, cloud and platform services, improving stability and integration capabilities.

The embodiment of the present invention also provides a data error detection device with a processor-memory architecture. FIG. 7 is an exemplary structural diagram of a data error detection device with a processor-memory architecture according to an embodiment of the present invention.

As shown in Figure 7, the data error detection device 700 includes a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the computer program is executed by the processor 701, any of the above can be achieved. Data error detection methods. Among them, the memory 702 can be implemented as various storage media such as electrically erasable programmable read-only memory (EEPROM), flash memory (Flash memory), programmable programmable read-only memory (PROM), etc. The processor 701 may be implemented to include one or more central processing units or one or more field programmable gate arrays, where the field programmable gate array integrates one or more central processing unit cores. Specifically, the central processing unit or central processing unit core may be implemented as a CPU, an MCU, a DSP, or the like.

It should be noted that not all steps and modules in the above-mentioned processes and structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The division of each module is only for the convenience of describing the functional division. In actual implementation, one module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located on the same device. , or it can be on a different device.

The hardware modules in various embodiments may be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuits or logic devices (such as a dedicated processor such as an FPGA or ASIC) to perform specific operations. Hardware modules may also include programmable logic devices or circuits (eg, including general-purpose processors or other programmable processors) temporarily configured by software to perform specific operations. As for the specific use of mechanical means, or the use of dedicated permanent circuits, or the use of temporarily configured circuits (such as configured by software) to implement the hardware modules, it can be decided based on cost and time considerations.

The above descriptions are only preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims (18)

1. A data error detection method, characterized in that the method includes:

   Determine (101) the data type contained in the template data;

   Determine (102) statistical characteristics of the data type based on statistics of values of the data type in the template data;

   Based on the data type and the statistical characteristics of the data type, determine (103) a structural file;

   Receive (104) test data;

   Compare the test data with the structure file (105);

   Based on the results of the comparison, it is determined (106) whether the test data is in error.
2. The data error detection method according to claim 1, characterized in that:

   Determining (106) whether the test data is in error based on the result of the comparison includes at least one of the following:

   When the data type contained in the test data does not match the data type in the structure file, it is determined that the test data is in error;

   When the value of the data type in the test data does not comply with the statistical characteristics in the structure file, it is determined that the test data is in error;

   When the value of the data type in the test data does not conform to the statistical characteristics in the structure file and the data type contained in the test data does not match the data type in the structure file, it is determined that the The above test data is wrong;

   The test is determined when the value of the data type in the test data conforms to the statistical characteristics in the structure file and the data type contained in the test data matches the data type in the structure file. The data is normal.
3. The data error detection method according to claim 2, further comprising:

   When the test data is erroneous, performing a correction operation on the test data based on the structure file;

   In the log file, data error events and the corrective actions are recorded.
4. The data error detection method according to claim 3, characterized in that, performing a correction operation on the test data based on the structure file includes at least one of the following:

   Based on the statistical characteristics of the data type contained in the structure file, correct the value of the corresponding data type in the test data;

   Based on the data type contained in the structure file, the data type in the test data is corrected.
5. The data error detection method according to claim 1, characterized in that determining (101) the data type contained in the template data includes:

   Determine the first data type in the template data and the object in the template data;

   Determine a second data type contained by the object;

   The first data type and the second data type are determined as data types included in the template data.
6. The data error detection method according to any one of claims 1-5, further comprising:

   Display the structure file on the human-computer interaction interface;

   Receive structural file adjustment instructions triggered via the human-computer interaction interface;

   Based on the structure file adjustment instruction, the data type in the structure file and/or the statistical characteristics in the structure file are adjusted.
7. The data error detection method according to any one of claims 1-5, characterized in that the statistical characteristics include at least one of the following:

   The maximum length of the data type; the minimum length of the data type; the maximum value of the data type; the minimum value of the data type; the average value of the data type; the maximum time interval of the data type; The minimum time interval for the above data type; the average time interval for the above data type.
8. The data error detection method according to any one of claims 1 to 5, characterized in that the template data is contained in an instance field of an application programming interface design file, or is included in an instance field obtained through real-time access to an application programming interface. in the real-time file.
9. The data error detection method according to any one of claims 1-5, characterized in that, before determining (101) the data type contained in the template data, the method further includes:

   Parse the template data from input data having multiple protocol encapsulation formats;

   Convert the template data into a predetermined data exchange format.
10. A data processing device, characterized in that the device includes:

    The first determination module (601) is configured to determine the data type contained in the template data;

    The second determination module (602) is configured to determine the statistical characteristics of the data type based on statistics of the values of the data type in the template data;

    The third determination module (603) is configured to determine the structure file based on the data type and the statistical characteristics of the data type;

    A receiving module (604) configured to receive test data;

    A comparison module (605) configured to compare the test data with the structure file;

    The fourth determination module (606) is configured to determine whether the test data is in error based on the result of the comparison.
11. The data processing device according to claim 10, characterized in that:

    The fourth determination module (606) is configured to perform at least one of the following:

    When the data type contained in the test data does not match the data type in the structure file, it is determined that the test data is in error;

    When the value of the data type in the test data does not comply with the statistical characteristics in the structure file, it is determined that the test data is in error;

    When the value of the data type in the test data does not conform to the statistical characteristics in the structure file and the data type contained in the test data does not match the data type in the structure file, it is determined that the The above test data is wrong;

    The test is determined when the value of the data type in the test data conforms to the statistical characteristics in the structure file and the data type contained in the test data matches the data type in the structure file. The data is normal.
12. The data processing device according to claim 11, further comprising:

    The correction module (607) is configured to perform a correction operation on the test data based on the structure file when the test data is erroneous; record the data error event and the correction operation in the log file.
13. The data processing device according to claim 12, characterized in that:

    The correction module (607) is configured to perform at least one of the following:

    Based on the statistical characteristics of the data type contained in the structure file, correct the value of the corresponding data type in the test data;

    Based on the data type contained in the structure file, the data type in the test data is corrected.
14. The data error detection device according to claim 10, characterized in that the first determination module (701) is configured to determine the first data type in the template data and the objects in the template data; determine The second data type contained in the object; determine the first data type and the second data type as the data types contained in the template data.
15. The data error detection device according to any one of claims 10-14, further comprising:

    The adjustment module (608) is configured to display the structure file on a human-computer interaction interface; receive a structure file adjustment instruction triggered via the human-computer interaction interface; and adjust the structure file based on the structure file adjustment instruction. the data type, and/or the statistical characteristics in the structure file.
16. An electronic device, characterized by including:

    processor(701);

    Memory (702), used to store executable instructions of the processor (701);

    The processor (701) is configured to read the executable instructions from the memory (702) and execute the executable instructions to implement the data error detection described in any one of claims 1-9 method.
17. A computer-readable storage medium on which computer instructions are stored, characterized in that when the computer instructions are executed by a processor, the data error detection method described in any one of claims 1-9 is implemented.
18. A computer program product, characterized by comprising a computer program that implements the data error detection method according to any one of claims 1-9 when executed by a processor.

Images