TW202217641A - Data checking method, device, equipment and system and storage medium - Google Patents

Abstract

The invention discloses a data checking method, device, equipment and system and a storage medium, and relates to the field of data processing. The method comprises the steps: under the condition that write operation occurs in all system data pools, generating and transmitting a data stream comprising data associated with the write operation, wherein the data stream comprises the main key value of the data; dividing the data stream into at least one data region based on the field of the data stream and a preset region division rule, wherein data region comprises the data stream corresponding to at least two system data pools; and in each data region, checking the data stream corresponding to the at least two system data pools in the data region according to the main key value corresponding to the data stream so as to determine whether the data of the at least two system data pools in the data region is consistent or not. According to the embodiment of the invention, the problem of cross-system data inconsistency can be discovered in time.

Description

Data verification method, device, equipment, system and storage medium

The present invention relates to the technical field of data verification, and in particular, to a data verification method, device, equipment, system and storage medium.

As the complexity of the business increases, a business will involve multiple systems. Correspondingly, among multiple systems, it is necessary to check business data across systems to find inconsistencies in business data across systems, so that measures can be taken for each system to ensure smooth operation of each system.

At this stage, business data can be obtained from the two systems involved in data verification. For example, obtain business data for the past 1 day from system A and system B, respectively, and compare the business data in system A and system B one by one to see if the business data in system A and system B are consistent, that is, whether there is a cross-system data inequity problem. However, this kind of data checking method cannot detect the data inequity across the system in time.

Embodiments of the present invention provide a data verification method, apparatus, device, system, and storage medium, which can timely discover the problem of data inequity across systems.

In a first aspect, an embodiment of the present invention provides a data verification method, including: when a write operation occurs in each system data pool, generating and transmitting a data flow including data associated with the write operation, where the data flow includes a primary key value of the data ; Based on the fields of the data flow and the preset area division rules, the data flow is divided into at least one data area, and each data area includes the data flows corresponding to at least two system data pools; in each data area, according to the data The primary key value corresponding to the process is checked against the data processes corresponding to at least two system data pools in the data area to determine whether the data of at least two system data pools in the data area are consistent.

In a second aspect, an embodiment of the present invention provides a data verification device, including: a data flow generation module, configured to generate a data flow including The data flow of the data associated with the write operation is transmitted and transmitted. The data flow includes the primary key value of the data; the area division module is used to divide the data flow into at least one data flow based on the fields of the data flow and the preset area division rules. area, each data area includes data processes corresponding to at least two system data pools; the check module is used to check at least two system data in the data area according to the primary key value corresponding to the data process in each data area The data flow corresponding to the pool is used to determine whether the data of at least two system data pools in the data area are consistent.

In a third aspect, an embodiment of the present invention provides a data verification device, including: a processor and a memory storing computer program instructions; the processor implements the data verification method of the first aspect when the processor executes the computer program instructions.

In a fourth aspect, an embodiment of the present invention provides a data verification system, including: a data flow device for generating and transmitting a data flow including data associated with the write operation when a write operation occurs in each system data pool, and the data flow Including the primary key value of the data; the diversion device is used to divide the data process into at least one data area based on the fields of the data process and the preset area division rules, and each data area includes the data corresponding to at least two system data pools Process; a checking device for checking the data processes corresponding to at least two system data pools in the data area according to the primary key value corresponding to the data process in each data area, so as to determine at least two system data in the data area Whether the pool data is consistent.

In a fifth aspect, an embodiment of the present invention provides a computer storage medium, where computer program instructions are stored thereon, and when the computer program instructions are executed by a processor, the data verification method of the first aspect is implemented.

Embodiments of the present invention provide a data verification method, apparatus, device, system, and storage medium, which generate a data flow including data associated with the write operation when a write operation occurs in each system data pool. The data process is divided into at least one data area, and each data area includes data processes corresponding to at least two system data pools. The data flow corresponding to the at least two system data pools is checked in the data area, so as to determine whether the data of the at least two system data pools are consistent. There is no need to set the time period for data acquisition, and the data process is triggered by the write operation, so as to divide and check the data process. Write operations are not limited by the length of time and can In the case of changes, the data is checked in real time, so that data inequities across the system can be found in time.

00,01,0001,0002,0003,0004,000792,000982,000991,000993: value

200: Data verification device

201: Data flow generation module

202: Regional division module

203: Check Module

2031: Window division unit

2032: Checking Unit

204: Screening Module

205: Processing modules

2008: Entrance

300: Data verification equipment

301: Memory

302: Processor

303: Communication interface

304: Busbar

41: Data flow device

42: shunt device

43: Checking device

A1,A2,A3: System

B1,B2,B3: system data pool

bussTp: transaction type

D1, D2, D3, D4: Check window

F3: Field

C1,C2,C3,C4: Data area

E1,E2,,L1,L2: Data flow

S101, S1011, S1012, S1013, S102, S103, S1031, S1032, S104, S105, S106: Steps

seqNo, traceId: primary key value

seqSt: business status

sysId: System ID

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following will briefly introduce the drawings that need to be used in the embodiments of the present invention. For those of ordinary skill in the art, without creative work, the Other schemas can be obtained from these schemas.

1 is a flowchart of an embodiment of a data verification method provided by the first aspect of the present invention;

FIG. 2 is a flowchart of another embodiment of the data verification method provided by the first aspect of the present invention

3 is a flowchart of another embodiment of the data verification method provided by the first aspect of the present invention;

FIG. 4 is a schematic diagram of an example of a check window of a data area according to an embodiment of the present invention

5 is a flowchart of still another embodiment of the data verification method provided by the first aspect of the present invention;

FIG. 6 is a schematic structural diagram of an embodiment of the data verification apparatus provided by the second aspect of the present invention;

7 is a schematic structural diagram of another embodiment of the data verification device provided by the second aspect of the present invention;

8 is a schematic structural diagram of another embodiment of the data verification device provided by the second aspect of the present invention;

FIG. 9 is a schematic structural diagram of still another embodiment of the data verification device provided by the second aspect of the present invention;

FIG. 10 is a schematic structural diagram of an embodiment of the data verification device provided by the third aspect of the present invention;

FIG. 11 is a schematic structural diagram of an embodiment of a data verification system according to a fourth aspect of the present invention.

The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present invention, but not to limit the present invention. It will be apparent to those skilled in the art that the present invention may be practiced without some of these specific details. under The embodiments are described only to provide a better understanding of the present invention by illustrating examples of the invention.

As the complexity of the business increases, a business may involve multiple systems, for example, a business is completed by the cooperation of multiple systems. In order to ensure the normal execution of the business, the systems involved in the business will store the data of the business, and check the data between multiple systems, so as to be able to find the inconsistency of the data between the multiple systems Therefore, measures can be taken to ensure the smooth operation of each system.

The amount of data that needs to be checked across systems is very large. Since the clocks of multiple systems may be different, in order to avoid missing the checked data, generally obtain business data within a long period of time, such as obtaining the completed day. The data of the respective businesses of multiple systems within the system are checked one by one to determine whether there is a data inequity problem between the systems. However, in this case, if there is a data inequity problem, the data inequity problem is only It can be found late, but not in time.

The present invention provides a data verification method, device, equipment, system and storage medium, which can transmit data in the form of data flow (ie Stream Data) when a write operation occurs, and utilize the primary key value of the data in the data flow , to check the data of different systems, so as to find the data inequity between the systems in time.

This does not limit the specific fields of business and data. For example, in the transaction field, the business may be transaction business, and the business data may be transaction flow data; the verification of the data is the transaction flow of the same transaction business. Data verification, the transaction details can be checked through data verification. However, the application scenarios of the embodiments of the present invention are not limited to transaction scenarios, and other application scenarios that require data verification are also within the protection scope of the embodiments of the present invention.

A first aspect of the present invention provides a data verification method. The data verification method can be performed by a data verification device, a data verification device or a data verification system. That is, the data verification method can be implemented by a single device or device, or by a system including multiple devices or devices. Implementation is not limited here.

FIG. 1 is an embodiment of the data verification method provided by the first aspect of the present invention flow chart. As shown in FIG. 1 , the data verification method may include steps S101 to S103 .

In step S101, when a write operation occurs in each system data pool, a data flow including data associated with the write operation is generated and transmitted.

The system data pool is used to store the data of the system, and specifically can be used to store the data of the business in the system. For example, in the transaction field, the system data pool can be used to store the flow data of the transaction business of the system. The system data pool may be set in the system, or may exist in the form of a database independently of the system, which is not limited herein. There can be multiple systems participating in data verification, and each system can correspond to a system data pool, that is, there can be multiple system data pools participating in data verification.

Write operations are operations that may cause changes to data in the system data pool. For example, write operations may include, but are not limited to, insert operations such as insert operations, update operations such as update operations, delete operations such as delete operations and drop operations, and create operations such as create operations. , modification operations such as alter operations, etc., are not limited here.

The data associated with the write operation includes the data on which the write operation was effected. A data flow is a collection of dynamic data that is not limited in time distribution and quantity. In the embodiment of the present invention, the data flow is used to carry the data. The data flow includes the primary key value of the data. The content of the primary key value of the data can be set according to the type of the data, which is not limited here. For example, the data includes transaction flow data, and the primary key value of the data may specifically include the transaction flow number.

In the case where the data check method is performed by the data check device or the data check device, the data flow may be transmitted within the data check device or the data check device. In the case where the data matching method is performed by a data matching system including a plurality of devices or devices, the data flow may be transferred between the devices or devices in the data matching system.

In step S102, the data flow is divided into at least one data region based on the fields of the data flow and a preset area division rule.

The fields of the data flow can be set according to the content and type of the data. For example, the fields of the data process may include a system identification field, a primary key value field, a business status field, etc., which are not limited herein. The system ID field is used to represent the ID of the system corresponding to the system data pool. The primary key value field is used to characterize the primary key value of the data. The business status field is used to represent the status of the business corresponding to the data. state.

According to the purpose of data verification, the data flow corresponding to each system data pool can be divided into multiple groups, that is, divided into at least one data area, according to the area division rules. Each data area includes data flows corresponding to at least two system data pools. The data included in the data flow can be checked in each data area. Each data area can correspond to the entry of the data process, and the division of the data process can be realized by setting the area division rules.

The area division rules can be set according to work scenarios and work requirements, and are not limited here. The data area can be regarded as a data process collection formed after the data process is grouped. The fields of the data flow of the same data area satisfy the same area division rule. In some examples, data checking is performed between systems, that is, data checking is performed between system data pools. Correspondingly, each data area may include data flows corresponding to two system data pools.

For example, a business involves three systems. The three systems are system A1, system A2, and system A3. The data of system A1 is stored in system data pool B1, the data of system A2 is stored in system data pool B2, and the data of system A3 is stored in the system data pool B3. When the data of the same business changes, under normal circumstances, the data of this business in the system data pool B1, system data pool B2 and system data pool B3 should all change; but there may be one or two of them. The situation that the data of this business in the system data pool has not changed is not limited here. The fields of the data flow can reflect the system identification, the primary key value of the data, the business status, etc. The data flow corresponding to the system data pool B1 and the data flow corresponding to the system data pool B2 can be divided into the data area C1 through the area division rules. The data flow corresponding to the system data pool B2 and the data flow corresponding to the system data pool B3 are divided into data areas C2. In the data area C1, the data check of the data process corresponding to the system data pool B1 and the data process corresponding to the system data pool B2 can be performed, and in the data area C2, the data process corresponding to the system data pool B2 and the system data pool B3 can be performed. Data verification of the data flow.

The data flow of a business corresponding to one system data pool may be divided into multiple data areas, or may be divided into one data area, which is not limited herein. For example, the zoning rule may limit the case where the value of field F3 of the data flow is one of 0001, 0002, and 0003 Next, the data flow is divided into the data area C3 through the entry 2008 . The area division rule can define that when the value of the field F3 of the data flow is one of 0003 and 0004, the data flow is divided into the data area C4 through the entry 2009 . The data flow with the value of 0003 in the field F3 will be divided into data area C3 and the data area C4; the data flow with the value of 0001 in the field F3 will be divided into the data area C3. The value of the field F3 of the data flow of the data area C3 satisfies the area division rule that the value of the field F3 of the data flow is one of 0001, 0002, and 0003. The value of the field F3 of the data flow of the data area C4 satisfies the area division rule that the value of the field F3 of the data flow is one of 0003 and 0004.

In step S103, in each data area, according to the primary key value corresponding to the data process, the data processes corresponding to at least two system data pools in the data area are checked to determine the data processes of the at least two system data pools in the data area. Are the data consistent.

Specifically, in each data area, the data flows corresponding to at least two system data pools with the same primary key value in the data area are checked. In the data area, there is a data flow corresponding to a system data pool whose primary key value is a certain value, but there is no data flow corresponding to another system data pool whose primary key value is a certain value. The data of at least two system data pools are inconsistent, that is, it is determined that a cross-system data inequity problem has occurred. In the data area, check the data of the data processes corresponding to at least two system data pools with the same primary key value. The data of at least two system data pools are consistent, that is, it is determined that there is no cross-system data inequity; The data in the system data pool is inconsistent, that is, it is determined that a cross-system data inequity problem has occurred.

In some examples, the checking of data flows in multiple data areas is performed concurrently. For example, after division, there are three data areas, namely, data area C1, data area C2, and data area C3. The verification of the data flow in the data area C1, the verification of the data flow in the data area C2, and the verification of the data flow in the data area C3 may be performed concurrently. The verification of data processes in multiple data areas is executed concurrently, which can speed up data verification and improve data verification. to efficiency. The verification of data flow in different data areas can be performed by different devices, equipment or modules, which is not limited herein. The data area can be increased or decreased according to specific needs, which improves the flexibility and scalability of data verification.

In some examples, the data flow check in each data area can be performed in the memory, so as to further improve the data check speed, improve the data check efficiency, and reduce the resource occupied by the data check.

In this embodiment of the present invention, when a write operation occurs in each system data pool, a data flow including data associated with the write operation is generated. The data process is divided into at least one data area, and each data area includes data processes corresponding to at least two system data pools. The data flow corresponding to the at least two system data pools is checked in the data area, so as to determine whether the data of the at least two system data pools are consistent. There is no need to set the time period for data acquisition, and the data process is triggered by the write operation, so as to divide and check the data process. The write operation is not limited by the length of time, and can check the data in real time when the data changes, so that the data inequity across the system can be found in time. Compared with the current method that takes a day or more to discover data inequities across systems, the data verification method provided by the embodiments of the present invention can shorten the time required to discover data inequities across systems to 1 minute or more. short.

In the case of a large amount of data, since the data generation process is triggered by the write operation, the data is checked in real time. Compared with the method of accumulating a large amount of data and then checking, it can meet higher performance requirements for data checking.

FIG. 2 is a flowchart of another embodiment of the data verification method provided by the first aspect of the present invention. The difference between FIG. 2 and FIG. 1 is that step S101 in FIG. 1 may be refined into steps S1011 to S1013 in FIG. 2 , and the data verification method shown in FIG. 2 may further include step S104 .

In step S1011, the binary log of each system data pool is read, and the write operation of each system data pool is determined according to the binary log.

The binary log is the BINLOG file, which is used to record the changes of the database table structure and the modification of table data. For example, the binary log records changes to the database table structure and Action statement for modification of table data. According to the contents of the binary log, the write operations that occurred in the system data pool can be determined.

In step S1012, based on the write operation, a data flow message is generated.

The data flow message is used to carry the data flow, and the specific format of the data flow message is not limited here. In some examples, the data flow message may specifically be a (JSON programming language, JavaScript object notation) JSON message. Use the data process message to carry the data process to facilitate the transmission of the data process. For example, the output format of a JSON message carrying a data flow is as follows:

Among them, sysId can represent the system ID, seqNo and traceId can represent the primary key value of the data at different stages, bussTp can represent the transaction type, and seqSt can represent the business status corresponding to the data.

Since the data corresponding to the same business may change, in order to enable the data flow to reflect the changes in the data, in some examples, the data flow message may include the main The data associated with the current write operation and the data associated with the last write operation with the same key value. Data with the same primary key value is the data corresponding to the same business. The data associated with the current write operation and the data associated with the previous write operation in the data flow message can reflect the change of the data, ensure that the correlation between the previous and previous data can be judged in the subsequent process, and according to the change of the data, determine whether to This data needs to be checked. For example, in the output format of the JSON message carrying the data flow described above, _before is used as the node label of the data associated with the last write operation and the data associated with the current write operation.

In step S1013, the data flow message is transmitted through the data flow element.

Data flow messages can be transmitted one by one through the data flow element. The data flow elements may include elements such as Kafka, which are not limited herein.

In some examples, before step S102 is performed, the data flow message may also be converted into a format that is more convenient for data verification. The execution of subsequent steps is convenient for data verification through configuration.

In step S104, in the case that a system data pool corresponds to a plurality of data flows with the same primary key value, a data flow in which a field meets a preset filter condition is reserved.

In the data process obtained based on the system data pool, there may be multiple data processes corresponding to one business, and it is necessary to filter the multiple data processes corresponding to one business, so that one data process corresponding to this business can participate in the data verification to avoid causing Data verification is confusing. The primary key value corresponding to the data process is the same, indicating that the business corresponding to the data process is the same business. Specifically, filter conditions can be set based on the meaning of each field of the data flow and the requirements for data verification, and one data flow can be selected and retained among multiple data flows with the same primary key value through the filter conditions. A data process whose reserved fields meet the filtering conditions can participate in the subsequent data verification process.

In some examples, the data flow includes a business status field. business status The field is used to represent the status of the business corresponding to the data of the data process. The above filtering conditions may include that the business status field includes the target value in the preset value set, and the business status field of the data flow is different from the business status field of the data flow corresponding to the last write operation. The preset value set includes at least one target value. The preset value set can be set according to work scenarios and work requirements, and is not limited herein.

For example, the value of the business status status column is 01, indicating that the data of the data flow does not need to be checked temporarily; the value of the business status status column is 00, indicating that the data of the data flow needs to be checked temporarily. The preset value set includes a target value of 00. When the business status field of the data flow L1 includes the target value of 00, and the business status field of the data flow corresponding to the last write operation is 01, the data flow L1 is reserved. When the business status field of the data flow L1 includes the target value of 00, but the business status field of the data flow corresponding to the last write operation is 00, the data flow L2 is discarded.

The filtering conditions are not limited to the above-mentioned contents, and the filtering conditions that can realize the filtering of multiple data processes with the same primary key value are all within the protection scope of the embodiments of the present invention, and will not be illustrated here one by one.

FIG. 3 is a flowchart of another embodiment of the data verification method provided by the first aspect of the present invention. The difference between FIG. 3 and FIG. 1 is that step S103 in FIG. 1 can be specifically refined into step S1031 and step S1032 in FIG. 3 .

In step S1031, in each data area, the data flow is divided into the check window according to the primary key value corresponding to the data flow.

The data flows in different check windows have different primary key values, that is, data flows with the same primary key value are not divided into different check windows, and data flows with the same primary key value are divided into the same check window. Dividing the data flow into check windows enables hashing of the data flow. In some examples, a certain check window of a certain data area includes data flows with the same primary key value corresponding to each system data pool corresponding to the data area. For example, the data area C1 includes the data flow corresponding to the system data pool B1 and the data flow corresponding to the system data pool B2, and a check window in the data area C1 may include a data flow corresponding to the system data pool B1 with the same primary key value and A data flow corresponding to the system data pool B2, that is, each data flow in the data area C1 In the check window, check the data flow of a pair of system data pool B1 and system data pool B2 with the same primary key value.

In step S1032, the data flow in the check window is checked.

Specifically, it is checked whether the data carried by the data flow in the check window are consistent. The fineness of the check window is smaller than that of the data area. In some cases, the data flow in the check window is triggered when the duration of the data flow in the check window memory exceeds the preset trigger time. In other cases, when the number of data flows in the verification window reaches a preset trigger number, the verification of the data flows in the verification window is triggered. Since the data flow in the embodiment of the present invention is triggered and generated by a write operation, and is not limited by the time length, the fineness of the check window can be very finely divided in terms of time or the number of data flows, thereby speeding up the data check speed. Improve data verification efficiency. Moreover, since the matching of the data flow has been completed in the process of dividing the data flow into the check window, the check of the data flow in the check window does not need to be matched, and can be implemented in a standardized and external program, which improves the flexibility of data check development and design The increase and decrease of the check window is relatively flexible and easy to expand.

In some examples, when the primary key value of the data flow in the existing check window is different from the primary key value corresponding to the undivided data flow, a new check window is generated, and the undivided data flow is divided into a new check window. in the check window. In the case that the time period of dividing the undivided data flow into the new check window exceeds the preset triggering time period, the check of the data flow in the new check window is triggered.

When the duration of the undivided data flow being divided into the new check window exceeds the preset trigger duration, and there is no data flow in the data area that can be checked with the data flow divided into the new check window, A data inconsistency problem may have occurred. The preset trigger duration can be set according to the work scenario and work requirements, and is not limited here. The preset trigger duration can be set by a timer. For example, when the timer count reaches the preset trigger duration, the verification of the data flow in the new verification window is triggered.

For example, FIG. 4 is a schematic diagram of an example of a check window of a data area according to an embodiment of the present invention. As shown in FIG. 4 , the existing check window in the data area C1 includes the check window Window D1, check window D2, and check window D3. The primary key value corresponding to the data process in the verification window D1 is 000792, the primary key value corresponding to the data process in the verification window D2 is 000982, and the primary key value corresponding to the data process in the verification window D3 is 000991. If the data flow E1 in the data area C1 has not been divided into the check window, and the primary key value corresponding to the data flow E1 is 000993, the primary key value of the data flow in the check window that already exists in the data area C1 corresponds to the data flow E1 The primary key values of , therefore, need to generate a new check window D4 for the data flow E1, and divide the data flow E1 into the check window D4. The preset trigger duration is set to 3 minutes. Correspondingly, 3 minutes after the data flow E1 is divided into the check window D4, the check of the data flow in the check window D4 is triggered.

In other examples, when the primary key value of the data flow in the existing check window is the same as the primary key value corresponding to the undivided data flow, the undivided data flow is divided into the existing check window. . When the number of data flows in the existing check window reaches the preset trigger quantity, the check of the data flows in the existing check window is triggered. Continue to wait when the number of data flows in the existing check window has not reached the preset trigger number.

The number of preset triggers can be set according to the work scenario and work requirements, and is not limited here.

For example, as shown in FIG. 4 , the existing check windows in the data area C1 include a check window D1 , a check window D2 and a check window D3 . The primary key value corresponding to the data process in the verification window D1 is 000792, the primary key value corresponding to the data process in the verification window D2 is 000982, and the primary key value corresponding to the data process in the verification window D3 is 000991. If the data flow E2 in the data area C1 has not been divided into the check window, and the primary key value corresponding to the data flow E2 is 000991, the data flow E2 is divided into the check window D3. The preset trigger number is set to 2. Correspondingly, when the number of data flows in the verification window D3 reaches 2, the verification of the data flows in the verification window D3 is triggered.

The verification of the data flow in the above embodiment can specifically check the value of the field of the data carried by the data flow, the number of the data flow in the verification window, etc., which are not limited herein.

FIG. 5 is a flowchart of still another embodiment of the data verification method provided by the first aspect of the present invention. The difference between FIG. 5 and FIG. 1 is that the data verification method shown in FIG. 5 may further include step S105 or step S106.

In step S105, when it is determined that the data of at least two system data pools in the data area are consistent, the value of the data verification success indicator is increased.

The data of at least two system data pools in the data area are consistent, that is, there is no cross-system data inequity problem, and the value of the data verification success indicator can be increased. The data verification success index is used to characterize the success rate of data verification. The larger the value of the data verification success indicator is, the higher the success rate of data verification is. The success index of data verification can provide a basis for cross-system data inequity, alarm, risk prediction, etc., and expand the application scope of data verification.

In step S106, when it is determined that the data of the at least two system data pools in the data area are inconsistent, the inconsistent data in the at least two system data pools in the data area is output.

The data of at least two system data pools in the data area are inconsistent, that is, a cross-system data inequity problem occurs, and the inconsistent data in at least two system data pools in the data area is the data that causes the cross-system data inequity problem. Inconsistent data in at least two system data pools in the data area can provide a basis for cross-system data inequity, alarms, risk prediction, etc., and expand the application scope of data verification.

It should be noted that, when the data verification method in the above-mentioned embodiment is executed by a data verification device or a data verification device, functions such as generating a data flow, dividing data areas, dividing a verification window, and data verification can be performed through different modules or unit implementation. When the data checking method in the above embodiment is executed by a data checking system, functions such as generating a data flow, dividing data regions, dividing checking windows, and checking data can be implemented by different devices. The specific form of the main body for executing the data verification method is not limited herein.

A second aspect of the present invention also provides a data verification device. FIG. 6 is a schematic structural diagram of an embodiment of the data verification apparatus provided in the second aspect of the present invention. As shown in FIG. 6 , the data verification device 200 may include a data flow generation module 201 , an area division module 202 and a verification module 202 . Module 203.

The data flow generation module 201 can be used to generate and transmit a data flow including data associated with the write operation when a write operation occurs in each system data pool.

Among them, the data flow includes the primary key value of the data.

The area division module 202 can be used to divide the data flow into at least one data area based on the fields of the data flow and the preset area division rule.

Wherein, each data area includes data processes corresponding to at least two system data pools.

In some examples, fields of data flows of the same data region satisfy the same region partitioning rules.

The checking module 203 can be used to check the data flows corresponding to at least two system data pools in the data region according to the primary key value corresponding to the data flow in each data region, so as to determine at least two system data pools in the data region are the data consistent.

In some examples, the checking of data flows in multiple data areas is performed concurrently.

In this embodiment of the present invention, when a write operation occurs in each system data pool, a data flow including data associated with the write operation is generated. The data process is divided into at least one data area, and each data area includes data processes corresponding to at least two system data pools. The data flow corresponding to the at least two system data pools is checked in the data area, so as to determine whether the data of the at least two system data pools are consistent. There is no need to set the time period for data acquisition, and the data process is triggered by the write operation, so as to divide and check the data process. The write operation is not limited by the length of time, and can check the data in real time when the data changes, so that the data inequity across the system can be found in time.

In some examples, the data flow generation module 201 can be used to: read the binary log of each system data pool, and determine the write operation of each system data pool according to the binary log; based on the write operation, generate a data flow message, data The process message is used to carry the data process; the data process message is transmitted through the data process element.

In some examples, the data flow message includes data associated with the current write operation and data associated with the previous write operation with the same primary key value.

FIG. 7 is a schematic structural diagram of another embodiment of the data verification apparatus according to the second aspect of the present invention. The difference between FIG. 7 and FIG. 6 is that the data verification apparatus 200 shown in FIG. 7 may further include a screening module 204 .

The filtering module 204 can be used for retaining a data process whose fields meet the preset filtering conditions when a system data pool corresponds to a plurality of data processes with the same primary key value.

In some examples, the data flow includes a business status field, and the business status field is used to represent the status of the business corresponding to the data of the data flow. The filtering conditions include: the business status field includes the target value in the preset value set, and the business status field of the data flow is different from the business status field of the data flow corresponding to the last write operation.

FIG. 8 is a schematic structural diagram of another embodiment of the data verification apparatus provided by the second aspect of the present invention. The difference between FIG. 8 and FIG. 6 is that the checking module 203 may include a window dividing unit 2031 and a checking unit 2032 .

The window dividing unit 2031 can be configured to divide the data flow into the check window according to the primary key value corresponding to the data flow in each data area.

The primary key value of the data flow in different check windows is different.

The checking unit 2032 can be used to check the data flow in the checking window.

In some examples, specifically, the window dividing unit 2031 can be configured to generate a new check window when the primary key value of the data flow in the existing check window is different from the primary key value corresponding to the undivided data flow, Divide undivided data flows into new check windows.

The checking unit 2032 can be configured to trigger checking of the data flow in the new checking window when the time period for which the undivided data flow is divided into the new checking window exceeds the preset triggering time period.

In other examples, specifically, the window dividing unit 2031 can be used to If the primary key value of the data flow in the existing check window is the same as the primary key value corresponding to the undivided data flow, divide the undivided data flow into the existing check window

The checking unit 2032 can be configured to trigger checking of the data flows in the existing checking window when the number of the data flows in the existing checking window reaches a preset trigger quantity.

FIG. 9 is a schematic structural diagram of still another embodiment of the data verification apparatus provided by the second aspect of the present invention. The difference between FIG. 9 and FIG. 6 is that the data verification apparatus 200 shown in FIG. 9 may further include a processing module 205 .

The processing module 205 can be configured to: in the case of determining that the data of at least two system data pools in the data area are consistent, increase the value of the data verification success indicator; when determining that the data of at least two system data pools in the data area are inconsistent In the case of at least two system data pools in the output data area that are inconsistent.

A third aspect of the present invention also provides a data verification device. FIG. 10 is a schematic structural diagram of an embodiment of the data verification device provided by the third aspect of the present invention. As shown in FIG. 10 , the data verification apparatus 300 includes a memory 301 , a processor 302 , and a computer program stored in the memory 301 and running on the processor 302 .

In one example, the above-mentioned processor 302 may include a central processing unit (Central Processing Unit, CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more of the embodiments of the present invention. multiple integrated circuits.

Memory may include Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical/tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (eg, memory devices) encoded with software including computer-executable instructions, and when the software is executed (eg, by one or more processors), it is operable to perform the operations described with reference to the data collation method according to the present invention.

The processor 302 runs a computer program corresponding to the executable code by reading the executable code stored in the memory 301, so as to implement the data checking method in the above-mentioned embodiment.

In one example, the data collation apparatus 300 may further include a communication interface 303 and a bus bar 304 . Among them, as shown in FIG. 10 , the memory 301 , the processor 302 , and the communication interface 303 are connected through the bus bar 304 to complete the communication with each other.

The communication interface 303 is mainly used to implement communication between modules, devices, units, and/or devices in the embodiments of the present invention. Input devices and/or output devices may also be accessed through the communication interface 303 .

The bus bar 304 includes hardware, software, or both, coupling the components of the data collation apparatus 300 to each other. By way of example and not limitation, the bus 304 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus , FSB), Hyper Transport (HT) interconnect, Industry Standard Architecture (ISA) bus, Infinite Bandwidth interconnect, Low pin count (LPC) bus, memory Body bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Extended (PCI-X) bus, serial Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local Bus (VLB) bus or other suitable bus or a combination of two or more of these . Busbars 304 may include one or more busbars, where appropriate. Although embodiments of the present invention describe and illustrate specific busbars, the present invention contemplates any suitable busbars or interconnects.

A fourth aspect of the present invention also provides a data verification system. FIG. 11 is a schematic structural diagram of an embodiment of a data verification system according to a fourth aspect of the present invention. As shown in FIG. 11 , the data verification system may include a data flow device 41 , a distribution device 42 and a verification device 43 . exist This does not limit the respective numbers of the data flow devices 41 , the distribution devices 42 and the verification devices 43 in the data verification system.

The data flow device 41 can be used to generate and transmit a data flow including data associated with the write operation when a write operation occurs in each system data pool.

The data flow includes the primary key value of the data.

The distribution device 42 can be used to divide the data flow into at least one data region based on the fields of the data flow and the preset region division rules.

Each data area includes data flows corresponding to at least two system data pools.

The checking device 43 can be used to check the data flow corresponding to at least two system data pools in the data region according to the primary key value corresponding to the data flow in each data region, so as to determine the data flow of the at least two system data pools in the data region. Are the data consistent.

The data flow device 41 , the distribution device 42 and the verification device 43 may also perform other steps in the data verification method in the above-mentioned embodiment. For details, please refer to the relevant description of the data verification method in the above-mentioned embodiment, which will not be repeated here.

A fifth aspect of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the data verification method in the above-mentioned embodiment can be implemented, and the same can be achieved. The technical effect, in order to avoid repetition, will not be repeated here. The above-mentioned computer-readable storage medium may include a non-transitory computer-readable storage medium, such as a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk etc., are not limited here.

It should be clear that each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment may be referred to each other, and each embodiment focuses on the differences from other embodiments. place. For the apparatus embodiments, device embodiments, system embodiments, and computer-readable storage medium embodiments, reference may be made to the description part of the method embodiments for related parts. The invention is not limited to the specific steps and structures described above and shown in the drawings. Those skilled in the art may make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present invention. Also, for the sake of brevity, it is omitted here Detailed description of known method techniques.

Aspects of the present invention are described above with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, or other programmable data processing device to produce a machine in which the instructions executed by the processor of the computer or other programmable data processing device cause the Implementations of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams can be performed. Such processors may be, but are not limited to, general purpose processors, special purpose processors, application specific processors, or field programmable logic circuits. It will also be understood that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can also be implemented by dedicated hardware for performing the specified functions or actions, or A combination of dedicated hardware and computer instructions is implemented.

Those skilled in the art should understand that the above-mentioned embodiments are all illustrative and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Those skilled in the art should be able to understand and implement other modified embodiments of the disclosed embodiments on the basis of studying the drawings, the description and the scope of the patent application. In the scope of the patent application, the term "comprising" does not exclude other devices or steps; the quantitative word "a" does not exclude a plurality of; the terms "first" and "second" are used to indicate names rather than any specific order. . Any drawing signs in the claims should not be construed as limiting the scope of protection. The functions of the multiple parts appearing in the request item may be implemented by a single hardware or software module. The presence of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

S101, S102, S103: Steps

Claims (15)

A data verification method, comprising: When a write operation occurs in each system data pool, a data flow including data associated with the write operation is generated and transmitted, and the data flow includes the primary key value of the data; Divide the data process into at least one data area based on the fields of the data process and a preset area division rule, and each of the data areas includes the data processes corresponding to at least two system data pools; In each of the data areas, according to the primary key value corresponding to the data process, the data processes corresponding to at least two system data pools in the data area are checked to determine at least two system data pools in the data area. Whether the data of each system data pool is consistent. The method according to claim 1, wherein, when a write operation occurs in each system data pool, generating and transmitting a data flow including data associated with the write operation includes: Read the binary log of each system data pool, and determine the write operation of each system data pool according to the binary log; Based on the write operation, a data flow message is generated, and the data flow message is used to carry the data flow; The data flow message is transmitted via the data flow element. The method according to claim 2, wherein the data flow message includes the data associated with the current write operation and the data associated with the previous write operation with the same primary key value. The method according to claim 1, wherein before dividing the data flow into at least one data region based on the fields of the data flow and a preset area division rule, the method further comprises: In the case that a system data pool corresponds to a plurality of the data flows with the same primary key value, one of the data flows whose fields meet the preset filter conditions is reserved. The method according to claim 4, wherein the data flow includes a business status field, and the business status field is used to represent the status of the business corresponding to the data of the data flow state, The screening conditions include: the business status field includes a target value in a preset value set, and the business status field of the data flow is different from the business status field of the data flow corresponding to the last write operation. The method according to claim 1, wherein, in each of the data areas, the data corresponding to at least two system data pools in the data area are checked according to the primary key value corresponding to the data process. Data flow, including: In each of the data regions, according to the primary key value corresponding to the data process, the data process is divided into a check window, and the primary key value of the data process in different check windows is different; Check the data flow in the check window. The method of claim 6, wherein, According to the primary key value corresponding to the data flow, the data flow is divided into the check window, including: In the case where the primary key value of the data flow in the existing check window is different from the primary key value corresponding to the undivided data flow, a new check window is generated, and the undivided data flow is into a new said check window; The checking of the data flow in the checking window includes: In the case that the time period for which the undivided data flow is divided into the new check window exceeds a preset trigger time period, the data flow in the new check window is checked for checking. The method of claim 6, wherein, According to the primary key value corresponding to the data flow, the data flow is divided into the check window, including: In the case that the primary key value of the existing data flow in the check window is the same as the primary key value corresponding to the undivided data flow, the undivided data flow is divided into the existing check in the window; The checking of the data flow in the checking window includes: When the number of the existing data flows in the verification window reaches a preset trigger number, triggering the verification of the existing data flows in the verification window. The method according to claim 1, wherein after checking the data flows corresponding to at least two system data pools in the data area according to the primary key value corresponding to the data flow, the method further comprises: In the case of determining that the data of at least two system data pools in the data area are consistent, increasing the value of the data verification success indicator; If it is determined that the data of the at least two system data pools in the data area are inconsistent, the inconsistent data in the at least two system data pools in the data area is output. The method of claim 1, wherein the fields of the data flow in the same data area satisfy the same area division rule. The method of claim 1, wherein the checking of the data flows in a plurality of the data areas is performed concurrently. A data verification device, characterized in that it includes: a data flow generation module, configured to generate and transmit a data flow including data associated with the write operation when a write operation occurs in each system data pool, and the data flow includes the primary key value of the data; The area division module is used to divide the data flow into at least one data area based on the fields of the data flow and the preset area division rules, and each of the data areas includes at least two system data pools corresponding to the data flow; A checking module, configured to check the data flows corresponding to at least two system data pools in the data area according to the primary key value corresponding to the data flow in each of the data areas, to determine the Whether the data of at least two system data pools in the data area are consistent. A data verification device, characterized in that it comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data verification method as described in any one of claims 1 to 11 is implemented. A data verification system, characterized in that it includes: a data flow device, configured to generate and transmit a data flow including data associated with the write operation when a write operation occurs in each system data pool, and the data flow includes a primary key value of the data; A flow dividing device, configured to divide the data flow into at least one data region based on the fields of the data flow and a preset area division rule, and each of the data regions includes the data corresponding to at least two system data pools. data flow; A checking device for checking the data flows corresponding to at least two system data pools in the data region according to the primary key value corresponding to the data flow in each of the data regions, so as to determine the data Whether the data of at least two system data pools in the region are consistent. A computer storage medium, characterized in that, computer program instructions are stored on the computer storage medium, and when the computer program instructions are executed by a processor, the data verification method described in any one of claim items 1 to 11 is implemented.

Images