CN115357625A - Structured data comparison method and device, electronic equipment and storage medium - Google Patents

Abstract

The application relates to the technical field of data processing, and provides a structured data comparison method, a device, an electronic device and a storage medium, wherein the method comprises the following steps: acquiring a structured data set and application scene requirements corresponding to the structured data set, wherein the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data; acquiring a field comparison rule based on application scene requirements, and analyzing the structured data set based on the field comparison rule to obtain comparison item numbers and sequencing fields; grouping the structured data sets based on the comparison item numbers to obtain at least one group of comparison data; and aiming at each group of comparison data, sequencing the comparison data based on the sequencing field, and sequentially comparing to obtain a comparison result. Therefore, the field comparison rules meeting the requirements of various application scenes can be supported, the comparison of batch data is supported, and the comparison efficiency is improved.

Description

Structured data comparison method, device, electronic device and storage medium

technical field

The present application relates to the technical field of data processing, and in particular to a structured data comparison method, device, electronic equipment and storage medium.

Background technique

Structured data, also known as row data, has specific fields, can be represented and stored using a relational database, and can be logically expressed in a two-dimensional table structure. When migrating structured data, it is necessary to Compare the data before and after the migration to ensure the correctness of the data migration.

In the prior art, when comparing structured data, the structured data file can be loaded into a relational database such as Oracle/Mysql, and the two tables can be associated through a full connection (SQL Full Join), and then the two tables Compare the strings in .

However, the above data comparison method depends on the performance of the stand-alone database. When there is batch data for comparison, it takes a lot of time and the comparison efficiency is poor.

Contents of the invention

The present application provides a structured data comparison method, device, electronic device, and storage medium, which can support field comparison rules that meet the requirements of various application scenarios, and simultaneously support batch data comparison to improve comparison efficiency.

In a first aspect, the present application provides a structured data comparison method, the method comprising:

Obtain a structured data set and application scenario requirements corresponding to the structured data set, the structured data set includes first structured data and second structured data, the first structured data is reference source data, and the The second structured data is changed data;

Obtaining field comparison rules based on the application scenario requirements, and parsing the structured data set based on the field comparison rules to obtain comparison item numbers and sorting fields;

grouping the structured data sets based on the comparison item numbers to obtain at least one set of comparison data;

For each set of comparison data, sort the comparison data based on the sorting field, and perform comparison in sequence to obtain a comparison result.

Optionally, the structured data set is parsed based on the field comparison rules to obtain the comparison item number and sorting fields, including:

Obtaining the corresponding comparison field in the field comparison rule, the comparison field includes a type field, a function field and a number field;

Obtain corresponding data in the structured data set based on the comparison fields, and preprocess the data based on the field comparison rules to obtain the comparison item number and sorting field corresponding to the data.

Optionally, the data is preprocessed based on the field comparison rule to obtain the comparison item number and sorting field corresponding to the data, including:

Identifying character strings to be compared in the data based on the type field, and splicing the character strings to be compared; the character strings to be compared include primary key character strings and business character strings;

Processing the spliced character strings to be compared based on the function field to obtain the compared character strings;

The comparison string is assembled by using the number field, and the comparison item number and sorting field corresponding to the assembled comparison string are identified.

Optionally, grouping the structured data set based on the comparison item number includes:

Obtain the primary key string corresponding to each data in the structured data set, and group the structured data set based on the primary key string and the comparison item number.

Optionally, sort the comparison data based on the sorting field, and perform comparison in sequence to obtain a comparison result, including:

Obtain the number of data items in each group of comparison data, and determine whether the number of data items is greater than 1;

If so, sorting the comparison data based on the sorting field, and performing comparisons in sequence to obtain a comparison result;

If not, the comparison result is obtained based on the source of the comparison data.

Optionally, the comparison result is obtained based on the source of the comparison data, including:

If the comparison data is from the first structured data, then determining that the comparison data is deleted data;

If the comparison data is from the second structured data, it is determined that the comparison data is newly added data.

Optionally, acquire the structured data set and the application scenario requirements corresponding to the structured data set, including:

Using a standardized interface to obtain real-time data in different databases, and converting the format of the real-time data to obtain a structured data set;

For the structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

Optionally, the method also includes:

After obtaining the comparison result, it is judged whether the comparison result is consistent;

If so, then compare the structured data set again at an interval of a preset time, and cover the last comparison result after obtaining the comparison result to verify the accuracy of the comparison;

If not, an alarm prompt is generated to remind the user to verify the comparison result.

In a second aspect, the present application provides a structured data comparison device, the device comprising:

An acquisition module, configured to acquire a structured data set and application scenario requirements corresponding to the structured data set, the structured data set includes first structured data and second structured data, and the first structured data is Reference source data, the second structured data is changed data;

A parsing module, configured to obtain field comparison rules based on the application scenario requirements, and analyze the structured data set based on the field comparison rules to obtain comparison item numbers and sorting fields;

A grouping module, configured to group the structured data set based on the comparison item number to obtain at least one set of comparison data;

The comparison module is configured to, for each group of comparison data, sort the comparison data based on the sorting field, and perform comparison in sequence to obtain a comparison result.

In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

the memory stores computer-executable instructions;

The processor executes the computer-implemented instructions stored in the memory to implement the method according to any one of the first aspects.

In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any one of the first aspect. described method.

In a fifth aspect, the present application provides a computer program product, including program code, and when a computer runs the computer program, the program code executes the method described in any one of the first aspect.

To sum up, the present application provides a structured data comparison method, device, electronic device and storage medium, which can obtain the structured data set and the application scenario requirements corresponding to the structured data set, and obtain the fields based on the application scenario requirements. The comparison rules, further, analyze the structured data set based on the field comparison rules to obtain the comparison item number and sorting field; then group the structured data set based on the comparison item number to obtain at least one set of comparison data ; Further, for each set of comparison data, sort the data based on the sorting field comparison, and perform comparison in turn to obtain the comparison result, wherein the structured data set includes the first structured data and the second structured data , the first structured data is the reference source data, and the second structured data is the changed data; in this way, it can support field comparison rules that meet the requirements of various application scenarios, and at the same time support the comparison of batch data. While improving the accuracy of data comparison, the comparison efficiency is improved.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description serve to explain the principles of the application.

FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

FIG. 2 is a schematic flow chart of a structured data comparison method provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of a framework of a standardized interface provided by an embodiment of the present application;

FIG. 4 is a schematic flow diagram of a batch data comparison method provided in an embodiment of the present application;

FIG. 5 is a schematic flow chart of a real-time data comparison method provided in an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a structured data comparison device provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

By means of the above drawings, specific embodiments of the present application have been shown, which will be described in more detail hereinafter. These drawings and text descriptions are not intended to limit the scope of the concept of the application in any way, but to illustrate the concept of the application for those skilled in the art by referring to specific embodiments.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present application as recited in the appended claims.

In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with basically the same function and effect. For example, the first device and the second device are only used to distinguish different devices, and their sequence is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the number and execution order, and words such as "first" and "second" do not necessarily limit the difference.

It should be noted that, in this application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design described herein as "exemplary" or "for example" is not to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In this application, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one item (piece) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

The application is introduced below with reference to the accompanying drawings. FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the application. A structured data comparison method provided by the application can be applied to the application scenario shown in FIG. 1 . The application scenario includes: the first database 1011, the second database 1012, the data processing platform 102 and the display terminal 103; when the data in the first database 1011 is migrated to the second database 1012, it is necessary to compare the data before and after the migration .

Specifically, the data processing platform 102 can acquire the reference source data from the first database 1011, and acquire the migrated data from the second database 1012, and correspondingly, acquire the configuration file corresponding to the application scenario, the configuration file is manual The field comparison rules applicable to the application scenario configured in advance, the configuration file includes the comparison fields and comparison rules that need to be compared with data, and further, the data processing platform 102 parses the configuration file to obtain the field comparison rules , and use this field comparison rule to compare the reference source data obtained from the first database 1011 with the migrated data obtained from the second database 1012 to obtain the comparison result, and compare the inconsistent The data is recorded, an alarm prompt is generated, and sent to the display terminal 103, so that the user operates the display terminal 103 to search and compare inconsistent data for verification.

It should be noted that the display terminal 103 may be a display device corresponding to the data processing platform 102, or may be a user's terminal device, which is not specifically limited in this embodiment of the application. The data in multiple databases can be compared, and the changed data in one database can also be compared. It is not limited to the data comparison in the data migration scenario of the two databases described above. The embodiment of this application The number of databases and application scenarios are not specifically limited.

The aforementioned terminal device may be a wireless terminal or a wired terminal. A wireless terminal may be a device that provides voice and/or other business data connectivity to a user, a handheld device with a wireless connection function, or other processing device connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via the radio access network (Radio Access Network, referred to as RAN), and the wireless terminal can be a mobile terminal, such as a mobile phone (or called a "cellular" phone) and a The computers, which may be, for example, portable, pocket, handheld, built-in or vehicle-mounted mobile devices, exchange speech and/or data with the radio access network. For another example, the wireless terminal may also be a Personal Communication Service (PCS for short) phone, a cordless phone, a Session Initiation Protocol (SIP for short) phone, a Wireless Local Loop (WLL for short) station , Personal Digital Assistant (Personal Digital Assistant, referred to as PDA) and other equipment. Wireless terminal can also be called system, subscriber unit (Subscriber Unit), subscriber station (Subscriber Station), mobile station (MobileStation), mobile station (Mobile), remote station (Remote Station), remote terminal (Remote Terminal), access A terminal (Access Terminal), a user terminal (User Terminal), a user agent (User Agent), and a user device (User Device or User Equipment) are not limited herein. Optionally, the aforementioned terminal devices may also be smart phones, tablet computers and other devices.

Therefore, comparing structured data can be applied to scenarios such as data migration or online maintenance transactions. Specifically, when migrating structured data, it is necessary to compare data before and after migration to ensure data migration correctness; when modifying data in an online maintenance transaction, it is necessary to see the difference before and after the data modification on the front end in real time, that is, it is necessary to record the difference before and after the data modification during the online maintenance transaction.

In a possible implementation, when comparing structured data, the structured data file can be loaded into a relational database such as Oracle/Mysql, and the two tables are associated through a full connection, and then the data in the two tables strings to compare.

However, the above data comparison method needs to load files into the database, and relies on the performance of a stand-alone database. When there is batch data for comparison, it takes a lot of time and the comparison efficiency is poor.

It should be noted that when there is batch data for comparison, multiple comparison rules are needed to meet different comparison requirements, but the implementation of complex comparison rules has a certain threshold for developers, and for real-time comparison Requirements, if implemented based on online transactions, need to be bound with transaction business logic, which will make online transactions very complicated.

In view of the above problems, this application provides a structured data comparison method, using Spark program development, that is, using distributed technology to replace stand-alone databases, loading and analyzing field comparison rules in different application scenarios through Spark programs, and then based on field comparison Analyze the structured data set by the rules to obtain the comparison item number and sorting field. The structured data set includes the reference source data and the changed data. Further, the structured data set is grouped by the comparison item number to obtain At least one set of comparison data, further, sort the comparison data in the same group based on the sorting field, and then perform data comparison, that is, put the comparison data of the same record and the same comparison item number together and sort them After comparison, it can support field comparison rules that meet the requirements of various application scenarios, and then support flexible comparison of batch data and improve comparison efficiency.

The technical solution of the present application will be described in detail below with specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below in conjunction with the accompanying drawings.

Fig. 2 is a schematic flow chart of a structured data comparison method provided in the embodiment of the present application. As shown in Fig. 2, the structured data comparison method includes the following steps:

S201. Acquire a structured data set and application scenario requirements corresponding to the structured data set, the structured data set includes first structured data and second structured data, and the first structured data is reference source data , the second structured data is changed data.

In this embodiment of the application, the first structured data and the second structured data may come from the same database or different databases, the structured data may be a database table or a data file, and the first structured data is the reference source data, that is, the database The original data stored in , and the second structured data is changed data, such as newly added, deleted, updated and migrated data, which is not specifically limited in this embodiment of the present application.

Optionally, the second structured data may be change data captured in real time by the real-time data change collection component from the redolog log, and the change data includes all fields of new, deleted, and updated records, then the first Structured data includes all fields recorded before adding, deleting, and updating, so that this application can collect real-time data and compare real-time data.

S202. Acquire field comparison rules based on the application scenario requirements, and analyze the structured data set based on the field comparison rules to obtain comparison item numbers and sorting fields.

In the embodiment of this application, the field comparison rules are configuration files defined in advance, and different application scenarios correspond to different configuration files. The configuration files include comparison fields and comparison rules, and the comparison fields are structured data sets The field type corresponding to the data in the data, such as field name, type field, function field and number field, etc., the comparison rule can be to use the obtained function field to process the data and then compare them sequentially, and/or use the number field to compare the data Combining and then sequentially comparing, and/or using data corresponding to a certain type of field to perform sequential comparison, etc., the embodiment of the present application does not limit the specific content corresponding to the comparison field and the comparison rule, which depends on the business scenario , and supports setting complex and diverse field comparison rules.

In this step, the structured data set is parsed based on the field comparison rule, that is, the corresponding comparison field in the field comparison rule is parsed out, and the structured data is obtained by using the comparison field The comparison item number and sorting field corresponding to the set, the comparison item number is used to mark the fields that need to be compared in the comparison data, such as 200010 means that the 9th to 10th fields in a piece of data need to be compared; The above-mentioned sorting fields may refer to fields that can be used for sorting in data, such as operator number, organization number, update timestamp, etc., which can be customized and designed by humans, which is not specifically limited in this embodiment of the application.

S203. Group the structured data set based on the comparison item number to obtain at least one set of comparison data.

In this step, the structured data set is grouped based on the comparison item number, that is, based on the comparison item number, the character strings to be compared are assembled, and further, the same record with the same comparison item The numbered comparison data are grouped into one group to obtain at least one group of comparison data.

S204. For each set of comparison data, sort the comparison data based on the sorting field, and perform comparison in sequence to obtain a comparison result.

In this step, the comparison data is sorted based on the sorting field, such as the data update timestamp corresponding to each group of structured data sets, and the data in each group is sorted in order based on the data update timestamp Sorting in order, and further, comparing the sorted data in turn to obtain the comparison result; the comparison result includes the result of consistent data and the result of inconsistent data, and for the result of inconsistent data, the application can also Inconsistent data is stored for subsequent queries.

Therefore, this application provides a method for comparing structured data, which can obtain the structured data set and the application scenario requirements corresponding to the structured data set, and obtain field comparison rules based on the application scenario requirements, and further, based on the field comparison The rules analyze the structured data set to obtain the comparison item number and sorting field; then group the structured data set based on the comparison item number to obtain at least one set of comparison data; further, for each set of comparison data , sort the data based on the sorting field comparison, and compare them in turn to obtain the comparison result, wherein the structured data set includes the reference source data and the changed data; thus, the structured data comparison method provided by this application It can support field comparison rules that meet the requirements of various application scenarios for data comparison, and at the same time support batch data comparison, which can improve the comparison efficiency while ensuring the correctness of data comparison.

Optionally, the structured data set is parsed based on the field comparison rules to obtain the comparison item number and sorting fields, including:

Obtaining the corresponding comparison field in the field comparison rule, the comparison field includes a type field, a function field and a number field;

Obtain corresponding data in the structured data set based on the comparison fields, and preprocess the data based on the field comparison rules to obtain the comparison item number and sorting field corresponding to the data.

In this embodiment of the application, the comparison field may include a type field, a function field, and a number field, and the type field is used to mark the type of field comparison, such as Y, N, K, S, C, HB, D, etc. Y indicates that this field is used for comparison, N indicates that this field is not used for comparison, K indicates that this field is used as a primary key, C indicates that this field is a fixed field, which can be customized by humans, and S indicates that this field corresponds to a comparison item number. HB indicates that the field participates in the comparison as a part of the joint field, and D indicates that the field is a sorting field; the function field can refer to a function expression for processing the field, such as trim, to_char, lower, etc., and the to_char indicates Convert the data type of the field to a character type. The trim means to delete the spaces in the data in the field. The trim means to convert the data in the field to a lowercase string. If there is no function field, it means no need Process the field; the number field is used to identify the comparison result of each business field, and the comparison result can be filtered according to the number, for example, the comparison item number of the 9th to 10th bytes of a certain data is 200010.

Exemplarily, the corresponding comparison fields in the field comparison rules in the embodiment of the present application can be shown in Table 1, and field comparison rules that meet the requirements of different application scenarios can be formed based on the comparison fields:

Table 1

Among them, the first column represents the English names of the fields corresponding to different fields in the data. The above is just an example, and the English names of the fields corresponding to the fields in the data can also be expressed in other forms, which are not specifically limited in this embodiment of the application. The second column represents the type field corresponding to the data, the third column represents the function field corresponding to the data, and the fourth column represents the number field corresponding to the data. The type field, the function field and the number field can be customized. This application The embodiment does not specifically limit this, and the above is only an example for illustration.

It can be understood that if the data in the structured data set corresponds to a function field, the function corresponding to the function field can be used to perform function processing on the field, and then the data processed by the function can be used for data comparison. Optionally, Commonly used string processing functions are preset in the field comparison rule, and the field comparison rule can also support user-defined functions to expand the function type corresponding to the function field, which is not specifically limited in this embodiment of the present application.

Therefore, the embodiment of the present application supports complex and diverse field comparison rules, and the data can be preprocessed by acquiring the comparison fields in the field comparison rules to improve processing flexibility.

Optionally, the data is preprocessed based on the field comparison rule to obtain the comparison item number and sorting field corresponding to the data, including:

Identifying character strings to be compared in the data based on the type field, and splicing the character strings to be compared; the character strings to be compared include primary key character strings and business character strings;

Processing the spliced character strings to be compared based on the function field to obtain the compared character strings;

The comparison string is assembled by using the number field, and the comparison item number and sorting field corresponding to the assembled comparison string are identified.

In the embodiment of this application, the primary key string can refer to the string data corresponding to the primary key, such as cst_id, and the business string refers to the string data corresponding to multiple business fields that make up a piece of data. For example, the contact address includes country and region , province, city, county, detailed address and other business fields, namely A country B province C city D county E village.

Optionally, if the data is composed of multiple business fields, separate each field with |@|, such as the data corresponding to the contact address: A country|@|B province|@|C city|@|D county| @|E Village, therefore, this application supports the combination of multiple business fields as one field for comparison, but if any one of the merged fields changes, the entire merged field needs to be used as the output result for data comparison. This application The embodiment does not specifically limit the delimiter used for combining each service field, and the above is only an example.

It is understandable that a piece of data can be split into multiple business fields for comparison by byte, so a certain business field in a piece of data can represent a business meaning, and other adjacent business fields can represent another business meaning.

In this step, identifying the character string to be compared in the data based on the type field can remove unnecessary comparison information, such as identifying the primary key field in the data and splicing the primary key string based on the identified primary key field; further , using the number field to assemble the comparison string, such as numbering each comparison item, assembling the string to be compared, and then identifying the number and sorting of the comparison item corresponding to the assembled comparison string field; among them, the fixed output field corresponding to the comparison string can also be identified, which is used for data comparison and improves the accuracy of the comparison.

Therefore, the embodiment of this application can support complex comparison logic, such as data can be split into multiple business fields for comparison, and supports functions to process data, and can also remove unnecessary comparison information, which is applicable to a wide range of scenarios.

Optionally, grouping the structured data set based on the comparison item number includes:

Obtain the primary key string corresponding to each data in the structured data set, and group the structured data set based on the primary key string and the comparison item number.

It can be understood that the present application may group the structured data set based on the primary key string.

Therefore, in the embodiment of the present application, the structured data set can be grouped by combining the primary key string and the comparison item number, or the structured data set can be grouped only based on the primary key string or the comparison item number, so as to improve grouping accuracy.

Optionally, sort the comparison data based on the sorting field, and perform comparison in sequence to obtain a comparison result, including:

Obtain the number of data items in each group of comparison data, and determine whether the number of data items is greater than 1;

If so, sorting the comparison data based on the sorting field, and performing comparisons in sequence to obtain a comparison result;

If not, the comparison result is obtained based on the source of the comparison data.

Exemplarily, in the application scenario of FIG. 1 , the reference source data obtained from the first database 1011 and the migrated data obtained from the second database 1012 are compared on the data processing platform 102 based on the comparison item number. Grouping, after obtaining at least one set of comparison data, the data processing platform 102 can obtain the number N of data pieces in each set of comparison data, and judge whether the number N of data pieces is greater than 1; if so, compare the data based on the sorting field Sorting and comparing in sequence to obtain the comparison result; if not, then obtain the comparison result based on whether the comparison data comes from the first database 1011 or the second database 1012, wherein, N is a positive integer greater than or equal to 1.

It can be understood that, in the embodiment of the present application, preset requirements can also be set to determine whether the preset requirements are met based on the number of data pieces, to determine whether the comparison data needs to be sorted and then compared, so as to obtain the comparison result. The embodiments of the present application do not specifically limit the preset requirements.

Therefore, the embodiment of the present application can sort and compare the data based on whether the number of data items in each group of comparison data meets the preset requirements, and then use the sorting field comparison to sort and compare the data to obtain the comparison result. When setting requirements, the comparison result can be directly determined according to the source of the comparison data to improve the comparison efficiency.

Optionally, the comparison result is obtained based on the source of the comparison data, including:

If the comparison data is from the first structured data, then determining that the comparison data is deleted data;

If the comparison data is from the second structured data, it is determined that the comparison data is newly added data.

In this step, if it is determined that the comparison data comes from the first structured data but does not exist in the second structured data, it means that the comparison data is deleted data; if it is determined that the comparison data If it comes from the second structured data but does not exist in the first structured data, it means that the comparison data is newly added data.

Therefore, the embodiment of the present application can directly obtain the comparison result based on the source of the comparison data, thereby improving the timeliness of the comparison.

Optionally, acquire the structured data set and the application scenario requirements corresponding to the structured data set, including:

Using a standardized interface to obtain real-time data in different databases, and converting the format of the real-time data to obtain a structured data set;

For the structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

In this embodiment of the application, the standardized interface may refer to a unified interface designed to call data in different databases. The standardized interface is a code framework defined in advance. When calling data in different databases, only need to modify the code framework corresponding parameter values.

In this step, the Kafka message middleware can be used to read the real-time change data of the database, but the results of parsing the change data by different database products are very different, so it is necessary to standardize the change data, and format the collected real-time data Transformation to obtain a structured data set that can be used for data comparison. The data format after standardized processing is unified, which facilitates the unification of subsequent processing procedures.

Among them, the data during the batch data comparison is in text format, and the standardized output of the real-time data comparison process is also in text format, and the text file formats of the two are consistent.

Exemplarily, FIG. 3 is a schematic diagram of a framework of a standardized interface provided by the embodiment of the present application; as shown in FIG. The data corresponds to the corresponding implementation class, and the call of each implementation class needs to be realized based on the standardized interface; the implementation classes corresponding to the changed data of databases such as Mysql, Oracle, DB2, OceanBase include: databaseType (data type), processinsertRec (increase), processDeleteRec (delete), processUpdateNotKeyRec (update non-primary key common fields), processUpdateKeyRec (update primary key field), are implemented based on standardized interfaces. Therefore, for various types of databases, users only need to implement this standardized interface.

It should be noted that different implementation classes in the calling database correspond to different comparison results. Taking the Oracle database as an example, the comparison results of each dml operation include transaction start and end marks, and there is only one result statement for adding/deleting; The comparison result corresponding to updating a non-primary key common field is to update the previous record or the updated record; the comparison result corresponding to updating the primary key field is the three statements V, D, and I. For the update primary key field of the OceanBase database, the corresponding comparison The result of the comparison is a delete operation, an insert operation, and other dml operations. The corresponding comparison result is a record containing information before and after the change. Different implementation classes in different databases correspond to different comparison results. The embodiment of this application is here I won't go into details one by one.

It is understandable that real-time comparison usually requires online maintenance transactions or a special back-end comparison program to perform customized comparisons for each changed data, which leads to high development and maintenance costs and a large amount of repetitive work. However, this application The embodiment can realize real-time data comparison based on a standardized interface, and also supports parsing data from different database products. Among them, the real-time comparison logic and batch comparison logic are consistent, the core code is shared, and the processing caliber is consistent, which can reduce later operation and maintenance costs. .

Optionally, the method also includes:

After obtaining the comparison result, it is judged whether the comparison result is consistent;

If so, then compare the structured data set again at an interval of a preset time, and cover the last comparison result after obtaining the comparison result to verify the accuracy of the comparison;

If not, an alarm prompt is generated to remind the user to verify the comparison result.

In the embodiment of the present application, the preset time may refer to the corresponding time set for re-comparison. The embodiment of the present application does not limit the specific value corresponding to the preset time, which may be one day, and the present application The embodiment does not specifically limit the content and sending form of the generated warning prompt. It can generate a warning prompt based on inconsistent data. For text messages and display boxes, etc.

It should be noted that although real-time data comparison makes up for the shortcoming of poor timeliness in batch processing, there is also the possibility that real-time data may be lost. Therefore, this application supports performing data comparison again after a preset time interval. After the comparison result is obtained, the last comparison result is overwritten, that is, the data reading application is insensitive to repeated comparisons through the method of database idempotence, thereby ensuring data timeliness and data integrity at the same time.

Therefore, the embodiment of the present application reduces the possible loss of real-time data through re-execution, improves data timeliness and improves data integrity, and can also generate an alarm prompt to remind the user of the abnormality of the comparison result, so as to facilitate Subsequent call data for verification to improve convenience.

In combination with the above-mentioned embodiments, the structured data comparison method provided by this application supports real-time and batch data comparison methods. Real-time comparison is based on flink datastream technology, batch comparison is based on flink dataset technology, real-time comparison logic and batch The comparison logic is consistent, the comparison caliber is consistent, and the two share the core code. Specifically, Fig. 4 is a schematic flow diagram of a batch data comparison method provided by the embodiment of the present application; as shown in Fig. 4, the batch data comparison The method includes the following steps:

Step 1: The Spark program starts, loads and parses the configuration file (application scenario requirements), and reads the data in file A and file B from the data file, and/or reads the data in table a and table b from the database table data, and parse the data according to the configuration file, go to step 2.

It should be noted that the data in file A and table a are reference source data, and the data in file B and table b are changed data. In this embodiment of the present application, the data in file A and file B can be compared separately The data in table a and table b can also be compared separately, or the data in file A and file B can be compared and the data in table a and table b can be compared separately. This embodiment of the present application does not specifically limit it.

Step 2: Merge the two parts of data in file A and file B, and/or, merge the two parts of data in table a and table b, and group the input data (structured data set) according to the primary key and comparison item number to obtain at least For a set of comparison data, the comparison data in the same group can be sorted according to the "data update timestamp", where the number of data items in the same group is recorded as N, and further, the comparison is determined according to the number of data items N. For the result, N is a positive integer greater than or equal to 1, and then go to step 3.

Step 3: If the number of data items is 1, and it is determined that the data comes from file A and/or table a, and there is no such data in file B and/or table b, it means that the record corresponding to the data is deleted. If it is determined that the data comes from If there is no such data in file B and/or table b, file A and/or table a, it means that the record corresponding to the data is newly added. If the number of data items is greater than 1, the sorted results can be compared in turn , to obtain the comparison result, wherein the data corresponding to the inconsistent item in the comparison can be recorded in the comparison result for the user to view.

Fig. 5 is a schematic flow chart of a real-time data comparison method provided in the embodiment of the present application; as shown in Fig. 5, the real-time data comparison method includes the following steps:

Step 1: Start the Spark Streaming program, load and parse the configuration file (application scenario requirements), and use the Kafka message middleware to call a unified standardized interface to collect the changed data and original data in the Oracle database in real time. In order to adapt to various database products, Standardize the change record into a json string in a unified format. Correspondingly, you also need to unify the value of the data update type, and adapt the record for updating the primary key, that is, convert the changed data and the original data into a unified format. Among them, the Oracle database After the data in is modified, it is called changed data, and then the subsequent steps are executed.

It should be noted that the subsequent execution steps are similar to step 2 and step 3 in the embodiment shown in FIG. 4 . For details, refer to the description of the embodiment shown in FIG. 4 above, and will not be described here again.

This application supports both real-time and batch methods for structured data comparison, both of which support complex and diverse field comparison rules in the form of configuration files. The threshold for learning and using is low, so that users do not need to develop repeatedly, and this application also foresees Commonly used string processing functions are set, custom functions are supported to extend function processing functions, and data comparison is performed by combining streams and batches, which reduces post-operation and maintenance costs and improves timeliness.

In the preceding embodiments, the structured data comparison method provided by the embodiments of the present application is introduced, and in order to realize the functions in the method provided by the above embodiments of the present application, the electronic device as the execution subject may include a hardware structure and/or Or a software module, the above-mentioned functions are realized in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above-mentioned functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

For example, FIG. 6 is a schematic structural diagram of a structured data comparison device provided in the embodiment of the present application. As shown in FIG. 6 , the device includes: an acquisition module 610, an analysis module 620, a grouping module 630, and a comparison module 640; Wherein, the acquiring module 610 is configured to acquire a structured data set and application scenario requirements corresponding to the structured data set, the structured data set includes first structured data and second structured data, and the first structured data set The first structured data is the reference source data, and the second structured data is the changed data;

The parsing module 620 is configured to obtain field comparison rules based on the application scenario requirements, and analyze the structured data set based on the field comparison rules to obtain comparison item numbers and sorting fields;

The grouping module 630 is configured to group the structured data set based on the comparison item number to obtain at least one set of comparison data;

The comparison module 640 is configured to, for each group of comparison data, sort the comparison data based on the sorting field, and perform comparison in sequence to obtain a comparison result.

Optionally, the parsing module 620 includes an acquisition unit and a preprocessing unit;

Specifically, the acquisition unit is configured to acquire the corresponding comparison field in the field comparison rule, and the comparison field includes a type field, a function field and a number field;

The preprocessing unit is configured to obtain corresponding data in the structured data set based on the comparison field, and preprocess the data based on the field comparison rules to obtain a comparison corresponding to the data Item number and sort fields.

Optionally, the preprocessing unit is specifically used for:

Identifying character strings to be compared in the data based on the type field, and splicing the character strings to be compared; the character strings to be compared include primary key character strings and business character strings;

Processing the spliced character strings to be compared based on the function field to obtain the compared character strings;

The comparison string is assembled by using the number field, and the comparison item number and sorting field corresponding to the assembled comparison string are identified.

Optionally, the grouping module 630 is specifically used for:

Obtain the primary key string corresponding to each data in the structured data set, and group the structured data set based on the primary key string and the comparison item number.

Optionally, the comparison module 640 includes a judgment unit, a comparison unit and a determination unit;

Specifically, the judging unit is used to obtain the number of data pieces in each group of comparison data, and judge whether the number of data pieces is greater than 1;

The comparison unit is configured to sort the comparison data based on the sorting field when the number of the data pieces is greater than 1, and perform comparisons in sequence to obtain a comparison result;

The determining unit is configured to obtain a comparison result based on a source of the comparison data when the number of data pieces is equal to 1.

Optionally, the determining unit is specifically used for:

If the comparison data is from the first structured data, then determining that the comparison data is deleted data;

If the comparison data is from the second structured data, it is determined that the comparison data is newly added data.

Optionally, the obtaining module 610 is specifically used for:

Using a standardized interface to obtain real-time data in different databases, and converting the format of the real-time data to obtain a structured data set;

For the structured data sets corresponding to different databases, obtain the application scenario requirements corresponding to each structured data set.

Optionally, the device further includes a judging module, and the judging module is configured to:

After obtaining the comparison result, it is judged whether the comparison result is consistent;

If so, then compare the structured data set again at an interval of a preset time, and cover the last comparison result after obtaining the comparison result to verify the accuracy of the comparison;

If not, an alarm prompt is generated to remind the user to verify the comparison result.

For the specific implementation principles and effects of a structured data comparison device provided in the embodiments of the present application, please refer to the relevant descriptions and effects corresponding to the above embodiments, and details will not be repeated here.

The embodiment of the present application also provides a schematic structural diagram of an electronic device. FIG. 7 is a schematic structural diagram of an electronic device provided in the embodiment of the present application. As shown in FIG. 7, the electronic device may include: a processor 701 and a The memory 702 is communicatively connected to the processor; the memory 702 stores a computer program; the processor 701 executes the computer program stored in the memory 702, so that the processor 701 executes the method described in any one of the above embodiments.

Wherein, the memory 702 and the processor 701 may be connected through a bus 703 .

The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium stores computer program execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the above-mentioned information in any of the foregoing embodiments of the application. Methods.

The embodiment of the present application also provides a chip for running instructions, and the chip is used to execute the method described in any of the foregoing embodiments executed by an electronic device in any of the foregoing embodiments of the present application.

The embodiment of the present application also provides a computer program product, the program product includes program code, and when the computer runs the computer program, the program code executes the aforementioned steps performed by the electronic device in any of the aforementioned embodiments of the present application. The method described in any of the Examples.

In the technical solution of this application, the collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data and other information involved are in compliance with relevant laws and regulations, and do not violate public order and good customs.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.

A module described as a separate component may or may not be physically separated, and a component shown as a module may or may not be a physical unit, that is, it may be located in one place, or may also be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to implement the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, each module may exist separately physically, or two or more modules may be integrated into one unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated modules implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium, and include several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) or a processor execute some steps of the methods described in various embodiments of the present application.

It should be understood that the above-mentioned processor may be a central processing unit (Central Processing Unit, referred to as CPU), and may also be other general-purpose processors, a digital signal processor (Digital Signal Processor, referred to as DSP), an application specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC) and so on. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in conjunction with the application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

The memory may include a high-speed random access memory (Random Access memory, referred to as RAM), and may also include a non-volatile memory (Non-volatile Memory, referred to as NVM), such as at least one disk memory, and may also be a U disk, a mobile hard disk , read-only memory, disk or CD-ROM, etc.

The bus may be an Industry Standard Architecture (Industry Standard Architecture, ISA for short) bus, a Peripheral Component Interconnect (PCI for short) bus, or an Extended Industry Standard Architecture (EISA for short) bus. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, the buses in the drawings of the present application are not limited to only one bus or one type of bus.

The above-mentioned storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as Static Random-Access Memory (Static Random-Access Memory, referred to as SRAM), electrically erasable programmable read-only Electrically Erasable Programmable Read Only Memory (EEPROM for short), Erasable Programmable Read-Only Memory (EPROM for short), Programmable Read-Only Memory (PROM for short), only Read-Only Memory (ROM for short), magnetic memory, flash memory, magnetic disk or optical disk. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.

An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be a component of the processor. The processor and the storage medium may be located in application specific integrated circuits (Application Specific Integrated Circuits, ASIC for short). Of course, the processor and the storage medium can also exist in the electronic device or the main control device as discrete components.

The above is only the specific implementation of the embodiment of the application, but the protection scope of the embodiment of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the application shall be covered by this application. Within the scope of protection of the application examples. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims (12)

1. A method for comparing structured data, the method comprising:

acquiring a structured data set and application scene requirements corresponding to the structured data set, wherein the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;

acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain a comparison item number and a sequencing field;

grouping the structured data set based on the comparison item numbers to obtain at least one group of comparison data;

and aiming at each group of comparison data, sequencing the comparison data based on the sequencing field, and sequentially comparing to obtain a comparison result.

2. The method of claim 1, wherein parsing the structured data set based on the field alignment rule to obtain an alignment number and a ranking field comprises:

acquiring a corresponding comparison field in the field comparison rule, wherein the comparison field comprises a type field, a function field and a number field;

and acquiring corresponding data in the structured data set based on the comparison field, and preprocessing the data based on the field comparison rule to obtain a comparison item number and a sequencing field corresponding to the data.

3. The method of claim 2, wherein preprocessing the data based on the field alignment rule to obtain an alignment number and a sorting field corresponding to the data comprises:

identifying character strings to be compared in the data based on the type fields, and splicing the character strings to be compared; the character strings to be compared comprise a main key character string and a service character string;

processing the spliced character strings to be compared based on the function fields to obtain comparison character strings;

and assembling the comparison character strings by using the number fields, and identifying comparison item numbers and sequencing fields corresponding to the assembled comparison character strings.

4. The method of claim 1, wherein grouping the structured data set based on the alignment numbers comprises:

and acquiring a primary key character string corresponding to each data in the structured data set, and grouping the structured data set based on the primary key character string and the comparison item number.

5. The method of claim 1, wherein sorting the comparison data based on the sorting field and sequentially comparing the sorted comparison data to obtain a comparison result comprises:

acquiring the number of data in each group of comparison data, and judging whether the number of the data is more than 1;

if yes, sorting the comparison data based on the sorting field, and sequentially comparing to obtain a comparison result;

if not, obtaining a comparison result based on the source of the comparison data.

6. The method of claim 5, wherein obtaining alignment results based on the source of the alignment data comprises:

if the comparison data is from the first structured data, determining the comparison data as deleted data;

and if the comparison data is from the second structured data, determining the comparison data as the newly added data.

7. The method of claim 1, wherein obtaining a structured data set and application scenario requirements corresponding to the structured data set comprises:

acquiring real-time data in different databases by using a standardized interface, and performing format conversion on the real-time data to obtain a structured data set;

and aiming at the structured data sets corresponding to different databases, acquiring the application scene requirements corresponding to each structured data set.

8. The method according to any one of claims 1-7, further comprising:

after the comparison result is obtained, judging whether the comparison result is consistent;

if yes, comparing the structured data set again at preset time intervals, and covering the last comparison result after obtaining the comparison result so as to verify the accuracy of the comparison;

and if not, generating an alarm prompt to remind the user to check the comparison result.

9. A structured data alignment apparatus, comprising:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a structured data set and application scene requirements corresponding to the structured data set, the structured data set comprises first structured data and second structured data, the first structured data is reference source data, and the second structured data is changed data;

the analysis module is used for acquiring a field comparison rule based on the application scene requirement, and analyzing the structured data set based on the field comparison rule to obtain a comparison item number and a sequencing field;

the grouping module is used for grouping the structured data set based on the comparison item numbers to obtain at least one group of comparison data;

and the comparison module is used for sequencing the comparison data based on the sequencing fields aiming at each group of comparison data and sequentially comparing the sequencing data to obtain a comparison result.

10. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored by the memory to implement the method of any of claims 1-8.

11. A computer-readable storage medium having computer-executable instructions stored thereon, which when executed by a processor, perform the method of any one of claims 1-8.

12. A computer program product comprising a program code for performing the method of any one of claims 1-8 when the computer program is run by a computer.

Images