CN112860722A - Data checking method and device, electronic equipment and readable storage medium - Google Patents
Data checking method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN112860722A CN112860722A CN202110312665.6A CN202110312665A CN112860722A CN 112860722 A CN112860722 A CN 112860722A CN 202110312665 A CN202110312665 A CN 202110312665A CN 112860722 A CN112860722 A CN 112860722A
- Authority
- CN
- China
- Prior art keywords
- data
- service
- checking
- group
- sets
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 30
- 238000012795 verification Methods 0.000 claims description 19
- 238000004458 analytical method Methods 0.000 claims description 13
- 238000004891 communication Methods 0.000 claims description 4
- 238000001514 detection method Methods 0.000 abstract description 4
- 230000008569 process Effects 0.000 description 13
- 238000004590 computer program Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 238000013524 data verification Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 5
- 238000004140 cleaning Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012958 reprocessing Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000012954 risk control Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data checking method, a data checking device, an electronic device and a readable storage medium, wherein a plurality of groups of service data are respectively acquired from a plurality of different data sources, each group of service data is data corresponding to different associated service items in the same service scene, each group of service data is processed to obtain a corresponding checking data set, and data in any two checking data sets is checked to detect whether the data in any two checking data sets meet the preset service consistency requirement. Therefore, the mode of checking and matching the processed core to the data set can be adopted, so that the service consistency detection among different service items under the same service scene can be realized, on one hand, the processing efficiency can be improved, the checking standard can be unified, and on the other hand, the consistency and the accuracy of the data can be ensured.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data checking method, an apparatus, an electronic device, and a readable storage medium.
Background
With the rapid development of internet technology, the internet now bears more and more complex business scenes, and the data is increased explosively. The fields of big data, BI (Business Intelligence) data, etc. have also grown up rapidly. In the current big data development process, data table cleaning is usually required to be completed according to data blood margins through a platform such as Hive. And the business or application layer can consume and use the data result of big data cleaning. How to ensure the integrity, consistency and accuracy of the cleaning data becomes a pain point for developing big data.
At present, no formed guarantee means exists in the aspect of big data quality guarantee, and for the quality guarantee of data, a big data development engineer generally guarantees the SQL quality and carries out SQL Review, so that the problems of low processing efficiency, non-uniform standard and the like exist.
Disclosure of Invention
The application aims to provide a data checking method, a data checking device, an electronic device and a computer readable storage medium, which can guarantee the consistency and the accuracy of data, improve the processing efficiency and unify the checking standard.
The embodiment of the application can be realized as follows:
in a first aspect, the present application provides a data collation method, including:
acquiring a plurality of groups of service data from a plurality of different data sources respectively, wherein each group of service data is data corresponding to different associated service items in the same service scene;
processing each group of service data to obtain a corresponding check data set;
and aiming at any two check data sets, checking the data in the two check data sets to detect whether the data in the two check data sets meet the preset service consistency requirement.
In an optional embodiment, the step of performing a reconciliation process on the data in the two reconciliation data sets to detect whether the data in the two reconciliation data sets satisfy a preset service consistency requirement includes:
checking whether the data of the set fields in the two checking data sets are consistent or not and whether the arrangement sequence of the data of the set fields is consistent or not;
and if the data of the set fields are consistent and the arrangement sequence is consistent, determining that the data in the two check data sets meet the preset service consistency requirement.
In an optional embodiment, the step of checking whether the data in the setting fields in the two data sets are consistent and the data in the setting fields are consistent in the arrangement order includes:
aiming at one of the two verification data sets, obtaining data under a first set field in the verification data set;
obtaining a second set field corresponding to the first set field in the other of the two verification data sets, and obtaining data under the second set field in the verification data set;
and checking whether the data in the first set field is consistent with the data in the second set field and whether the data arrangement sequence is consistent.
In an optional implementation manner, each group of service data is obtained based on a service logic module which is written in advance, and each group of service logic module contains an SQL statement;
the step of processing each group of service data to obtain a corresponding check data set includes:
syntax analysis is carried out on each group of service logic modules so as to detect whether the syntax of each group of service logic modules is correct or not;
and if the grammar of each group of business logic modules is correct, obtaining a corresponding check data set according to the SQL sentences in each group of business logic modules and the corresponding business data.
In an optional embodiment, the step of obtaining a corresponding check data set according to the SQL statements and the corresponding service data in each group of service logic modules includes:
obtaining a service data set according to SQL sentences in each group of service logic modules and corresponding service data;
and when the expansion requirement of the data of the specific field in the service data set is obtained, acquiring the expansion data corresponding to the data under the specific field, and adding the expansion data into the service data set to obtain the check data set.
In an optional embodiment, the step of obtaining a service data set according to the SQL statements and corresponding service data in each group of service logic modules includes:
analyzing SQL sentences in each group of business logic modules by adopting a Durid middleware to obtain an analysis result;
and combining the analysis result and the acquired service data to obtain a corresponding service data set.
In an alternative embodiment, the method further comprises:
when the data in any two check data sets are determined not to meet the preset service consistency requirement, generating corresponding prompt information;
and sending the prompt information to a user subscribing the check event.
In a second aspect, the present application provides a data collation apparatus, said apparatus comprising:
the acquisition module is used for respectively acquiring a plurality of groups of service data from a plurality of different data sources, wherein each group of service data is data corresponding to different associated service items in the same service scene;
the processing module is used for processing each group of service data to obtain a corresponding check data set;
and the checking module is used for checking the data in any two checking data sets so as to detect whether the data in the two checking data sets meet the preset service consistency requirement.
In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing processor-executable machine-executable instructions that, when executed by the electronic device, are executed by the processors to perform the method steps of any one of the preceding embodiments.
In a fourth aspect, the present application provides a computer-readable storage medium having stored thereon machine-executable instructions which, when executed, implement the method steps of any one of the preceding embodiments.
The beneficial effects of the embodiment of the application include, for example:
the data checking method, the data checking device, the electronic device and the readable storage medium provided by the embodiment of the application acquire a plurality of groups of service data from a plurality of different data sources respectively, wherein each group of service data is data corresponding to different associated service items in the same service scene, then process each group of service data to obtain a corresponding checking data set, and check the data in any two checking data sets to detect whether the data in any two checking data sets meet a preset service consistency requirement. Therefore, the mode of checking and matching the processed core to the data set can be adopted, so that the service consistency detection among different service items under the same service scene can be realized, on one hand, the processing efficiency can be improved, the checking standard can be unified, and on the other hand, the consistency and the accuracy of the data can be ensured.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.
Fig. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure;
FIG. 2 is a flowchart of a data checking method according to an embodiment of the present application;
FIG. 3 is a flowchart of sub-steps included in step S120 of FIG. 2;
FIG. 4 is a flowchart of sub-steps included in step S122 of FIG. 3;
FIG. 5 is a flowchart of sub-steps included in step S130 of FIG. 2;
FIG. 6 is a flowchart of sub-steps included in step S131 in FIG. 5;
fig. 7 is a functional block diagram of a data verification apparatus according to an embodiment of the present application.
Icon: 110-a processor; 120-a memory; 130-data collation means; 131-an acquisition module; 132-a processing module; 133-collation module.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
In the description of the present application, it should be noted that the features in the embodiments of the present application may be combined with each other without conflict.
As shown in fig. 1, an electronic device may be provided in the embodiments of the present application, and may be used to perform matching on data of different data sources. The electronic device may be a server or a terminal device, such as a personal computer.
The electronic device may include a memory 120 and a processor 110, and the memory 120 may have a data checking device 130 disposed therein.
In detail, the memory 120 and the processor 110 are electrically connected directly or indirectly to realize data transmission or interaction. For example, they may be electrically connected to each other via one or more communication buses or signal lines. The data collating means 130 includes at least one software function module which may be stored in the memory 120 in the form of software or firmware (firmware). The processor 110 is configured to execute an executable computer program stored in the memory 120, for example, a software functional module and a computer program included in the data verification apparatus 130, so as to implement the data verification method provided by the embodiment of the present application, thereby completing verification and matching of data of different data sources.
The Memory 120 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like.
The Processor 110 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), a System on Chip (SoC), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.
It will be appreciated that the arrangement shown in fig. 1 is merely illustrative and that the electronic device may also comprise more or fewer components than shown in fig. 1 or have a different configuration than shown in fig. 1, e.g. may also comprise a communication unit for information interaction with other devices.
It should be noted that, when the electronic device is used for performing check batch matching on data of different data sources, a specific purpose is not limited, and may be selected according to actual application requirements.
For example, in an alternative example, it may be that in a banking scenario, a big data developer may produce a daily loanable balance for a bank by writing SQL, and there is also a big-out-of-bank-data application that may expose the daily loanable balance for each bank. In order to ensure the accuracy of the loanable balance, a checking implementation module can be written in the electronic device to check the produced loanable balance and the application-displayed loanable balance, so that the business consistency of data obtained from different data sources is guaranteed.
With reference to fig. 2, an embodiment of the present application further provides a data checking method applicable to the electronic device. Wherein the method steps defined by the flow related to the data verification method can be implemented by the electronic device. The specific process shown in fig. 2 will be described in detail below.
Step S110, obtaining multiple sets of service data from multiple different data sources, wherein each set of service data is data corresponding to different associated service items in the same service scenario.
Step S120, processing each group of service data to obtain a corresponding check data set.
Step S130, for any two check data sets, performing a check process on data in the two check data sets to detect whether the data in the two check data sets meet a preset service consistency requirement.
In this embodiment, collation of data of multiple data sources can be realized based on the drive, which is an efficient data query system and can realize aggregate query of a large amount of time-series-based data. The different data sources may be different associated business items in the same business scenario, such as business items related to the loan amount information of the user, business items related to risk management and control of the user, and the like in a banking business scenario.
In this embodiment, the number of the data sources may be at least two, and the service data acquired from different data sources may be at least two groups. Each acquired group of business data may be specific information data under a corresponding business project, for example, the acquired group of business data may include loan information of multiple users, such as names, identification numbers, loan time, loan amount, and the like of loan users. In addition, the acquired group of service data may further include risk management and control information of the user, such as a name, an identification number, time, and the like of the user who passes the wind control.
In this embodiment, the data checking may be implemented in an offline scenario, for example, the service data in the preset duration may be acquired every preset duration, and then the checking processing is implemented offline based on the acquired service data. The preset duration may be, for example, one day, two days, one week, or the like, and may be specifically set according to actual needs.
As can be seen, the obtained service data may be specific information data within a period of time, and thus, each group of service data may be processed, for example, for each group of service data, the service data may be summarized, arranged, and the like, so as to obtain a corresponding check data set.
Alternatively, the resulting collated data set may be presented in the form of a list.
As can be seen from the above, the data from different data sources may be data corresponding to different service items with associations, and under the same service scenario, the data of different service items should be corresponding and matchable. For example, regarding a business item of the user loan information and a business item of the user risk management, the user who completed the loan should be risk-managed-passed, and thus, the user information of the user who completed the loan should be the same as the user information of the user who passed the risk management.
Therefore, for the obtained plurality of check data sets, the data in any two of the check data sets can be checked, so as to detect whether the data in the two check data sets meet the preset service consistency requirement.
When the data under different business projects meet the preset business consistency requirement, for example, taking the above as an example, the user information of the user who completes the loan within a period of time is the same as the user information of the user who passes the risk management and control within a period of time, which indicates that the data of the two business projects are consistent. Thus, the consistency and the accuracy of the data can be guaranteed through the codes.
In this embodiment, after the service data collected from different data sources is processed to obtain corresponding check data sets, the check data sets are checked and matched, so that the consistency of the data under different service items is detected.
In this embodiment, each group of service data may be obtained based on a service logic module compiled in advance, each group of service logic module includes an SQL statement, and optionally, referring to fig. 3, when each group of service data is processed to obtain a corresponding check data set, the following method may be implemented:
step S121, parsing the syntax of each group of service logic modules to detect whether the syntax of each group of service logic modules is correct, and if the syntax of each group of service logic modules is correct, performing the following step S122.
And step S122, obtaining a corresponding check data set according to the SQL sentences in each group of business logic modules and the corresponding business data.
In this embodiment, the service logic module may use, for example, ES5 grammar, java grammar, python, or the like as the code logic grammar. First, whether a syntax error exists in the business logic code written by the tester can be detected, for example, an ES5 syntax parser, a java syntax parser or a python syntax parser can be used to detect accordingly. If the grammar of the business logic module is detected to have problems, prompt information can be generated, and testers can be reminded to modify the grammar.
When the syntax of the business logic module has no problem, the syntax analysis can be performed on the SQL statements written in the business logic module, for example, the dual SQL parser can be used for performing the SQL syntax analysis, thereby ensuring the correctness of the SQL syntax.
After the syntax detection is passed, the obtained business data can be processed based on the SQL statements written in the business logic module to obtain the corresponding check data set. For example, the business data may be classified, summarized, sorted, and the like.
In the prior art, when data is checked, the data obtained after preliminary processing may not meet the requirements of business consumers, or corresponding comparison items do not exist in two sets of data sets for checking and matching, thereby causing checking obstacles. In view of this, referring to fig. 4, in this embodiment, when obtaining the check data set based on the SQL statement and the acquired data, the following steps may be performed:
and step S1221, obtaining a service data set according to the SQL statements and the corresponding service data in each group of service logic modules.
Step S1222, when an expansion requirement for data of a specific field in the service data set is obtained, obtaining expansion data corresponding to the data in the specific field, and adding the expansion data to the service data set to obtain the check data set.
In this embodiment, the acquired service data is processed to obtain a preliminary service data set. The obtained service data set may be in the form of a list, wherein the obtained service data set includes a plurality of fields, each of which may include a plurality of related data, for example, the fields may be fields such as user name, user ID, time, amount of money, etc.
In a possible scenario, if the obtained service data set does not include the identity card number of the user, however, the service consumer may need to use the identity card number of the user when applying the service data set subsequently, or the service data set used for comparing with the service data set includes the identity card number, and the identity card number needs to be used for checking and matching subsequently.
For the above scenario, in this embodiment, when an expansion requirement for data of a specific field to which a service data set belongs is obtained, expansion data corresponding to data in the specific field may be obtained and added to the service data set to form a checking data set for checking.
The specific field may be any field in a specified service data set, such as the user name field described above, and the obtained extension data may be obtained from the outside, obtained from a database, and the like, without limitation, such as the information data of the identity card described above.
In this way, by reprocessing the obtained service data set, a data set capable of meeting the specific requirements of the service consuming end can be obtained.
In this embodiment, when the business data is processed according to the SQL statements to obtain the business data set, the Durid middleware may be first adopted to analyze the SQL statements in each group of business logic modules to obtain an analysis result. Before this step, the syntax of the SQL statement may also be detected in advance to determine whether there is an error in the syntax of the SQL statement, and if there is no error, the parsing result is obtained by parsing through the Durid middleware. And then combining the analysis result and the acquired service data to obtain a corresponding service data set.
As described above, after the collation data sets corresponding to the service data are obtained, the collation processing can be performed on any two collation data sets. Referring to fig. 5, in the present embodiment, the checking process can be implemented as follows:
step S131, checking whether the data of the setting fields in the two checking data sets are consistent and whether the arrangement sequence of the data of the setting fields is consistent.
Step S132, if the data in the set fields are consistent and the arrangement order is consistent, determining that the data in the two check data sets meet a preset service consistency requirement.
In this embodiment, each of the obtained collation data sets may include a plurality of fields, such as the user name, the user ID, and the loan amount. By way of example, when the loan user information and the user information managed by the risk need to be checked, the setting field may be, for example, the user name.
That is, for two check data sets to be checked, data under the name field of the user belonging to the two check data sets can be searched respectively, and the data under the field of the two check data sets are compared. If the data under the field in the two verification data sets are consistent, all users completing the loan are indicated to be risk-controlled, and business consistency is maintained between the loan business and the risk-control business.
On this basis, it is considered that the data in the collation data set are arranged in chronological order of generation, and therefore, the position of each data in the collation data set indicates chronological order of generation thereof. In order to detect whether or not the data in the two collation data sets temporally coincide with each other in the time dimension, it is necessary to detect whether or not the data in the setting fields belonging to the two collation data sets coincide with each other in the order of arrangement of the data in the setting fields belonging to the two collation data sets.
If the data of the setting fields in the two verification data sets are consistent and the arrangement sequence is consistent, the data in the two data sets can be kept consistent on the time axis. In this case, it may be determined that the data in the two collated data sets satisfy a preset business consistency requirement.
In this embodiment, it is considered that there is a possibility that the setting manner of the fields in different collation data sets is different, and for example, the fields to which data actually containing the same attribute information may belong in different collation data sets are different, so that it is difficult to directly perform collation. In view of this, referring to fig. 6, in the present embodiment, when performing the verification process based on the data in the setting field, the verification process can be implemented by:
step S1311 is to obtain, for one of the two collation data sets, data in the first setting field in the collation data set.
Step S1312 obtains a second setting field corresponding to the first setting field in the other of the two collated data sets, and obtains data in the second setting field in the collated data set.
Step S1313, checking whether the data in the first setting field and the data in the second setting field are consistent, and whether the data arrangement order is consistent.
In this embodiment, for two checking data sets to be checked, by acquiring data in a first setting field and data in a second setting field corresponding to the two checking data sets, and comparing the consistency and the ordering order of the two sets of acquired data, it is determined whether the data in the two checking data sets meet a preset service consistency requirement.
In this embodiment, when it is determined that the data in any two of the check data sets does not satisfy the preset service consistency requirement, the corresponding prompt information may be generated, and the prompt information may be sent to the user subscribing to the check event. For example, the user who subscribes to the collation may be notified in the form of mail.
In this embodiment, in addition to the above-described manner of collating data sets, it is also possible to detect each collation data set by setting an assertion condition in advance, for example, the set assertion condition may be as follows: the check data set is not empty, or is as follows: the number of the data in the check data set is greater than or equal to 100, and the like, and the embodiment is not limited in particular, and may be set according to actual requirements.
In this way, when it is determined that the data in any two verification data sets do not meet the preset service consistency requirement or do not meet the preset requirement, the operation of generating the prompt message to send to the user subscribing to the verification event can be triggered.
The data checking scheme provided by this embodiment is an offline data checking mode based on Durid, and can guarantee data quality for big data under a distributed service framework. By generating the check data set and performing check processing on the check data set, the consistency, accuracy and integrity of data under the associated business items can be guaranteed. The risk of data errors is greatly reduced, and the problems of resource loss, online risk and the like caused by the data errors are solved.
With reference to fig. 7, an embodiment of the present application further provides a data verification apparatus 130 applicable to the electronic device. The data checking device 130 may include an obtaining module 131, a processing module 132 and a checking module 133.
The obtaining module 131 is configured to obtain multiple sets of service data from multiple different data sources, where each set of service data is data corresponding to different associated service items in the same service scenario.
In this embodiment, the obtaining module 131 may be configured to execute step S110 shown in fig. 2, and reference may be made to the foregoing description of step S110 for relevant contents of the obtaining module 131.
The processing module 132 is configured to process each group of service data to obtain a corresponding check data set.
In this embodiment, the processing module 132 may be configured to execute step S120 shown in fig. 2, and reference may be made to the foregoing description of step S120 for relevant contents of the processing module 132.
The checking module 133 is configured to, for any two checking data sets, perform checking processing on data in the two checking data sets to detect whether the data in the two checking data sets meet a preset service consistency requirement.
In this embodiment, the checking module 133 can be used to execute step S130 shown in fig. 2, and reference may be made to the foregoing description of step S130 for relevant contents of the checking module 133.
In a possible implementation manner, the checking module 133 may specifically be configured to:
checking whether the data of the set fields in the two checking data sets are consistent or not and whether the arrangement sequence of the data of the set fields is consistent or not;
and if the data of the set fields are consistent and the arrangement sequence is consistent, determining that the data in the two check data sets meet the preset service consistency requirement.
In a possible implementation manner, the checking module 133 may specifically be configured to:
aiming at one of the two verification data sets, obtaining data under a first set field in the verification data set;
obtaining a second set field corresponding to the first set field in the other of the two verification data sets, and obtaining data under the second set field in the verification data set;
and checking whether the data in the first set field is consistent with the data in the second set field and whether the data arrangement sequence is consistent.
In a possible implementation manner, each group of service data is obtained based on a service logic module written in advance, each group of service logic module includes an SQL statement, and the processing module 132 may be specifically configured to:
syntax analysis is carried out on each group of service logic modules so as to detect whether the syntax of each group of service logic modules is correct or not;
and if the grammar of each group of business logic modules is correct, obtaining a corresponding check data set according to the SQL sentences in each group of business logic modules and the corresponding business data.
In a possible implementation manner, the processing module 132 may specifically be configured to:
obtaining a service data set according to SQL sentences in each group of service logic modules and corresponding service data;
and when the expansion requirement of the data of the specific field in the service data set is obtained, acquiring the expansion data corresponding to the data under the specific field, and adding the expansion data into the service data set to obtain the check data set.
In a possible implementation manner, the processing module 132 may specifically be configured to:
analyzing SQL sentences in each group of business logic modules by adopting a Durid middleware to obtain an analysis result;
and combining the analysis result and the acquired service data to obtain a corresponding service data set.
In a possible implementation manner, the data checking apparatus 130 may further include an information generating module, and the information generating module may be specifically configured to:
when the data in any two check data sets are determined not to meet the preset service consistency requirement, generating corresponding prompt information;
and sending the prompt information to a user subscribing the check event.
In an embodiment of the present application, there is also provided a computer-readable storage medium, in which a computer program is stored, and the computer program executes the steps of the data checking method.
The steps executed when the computer program runs are not described in detail herein, and reference may be made to the explanation of the data checking method above.
To sum up, the data checking method, the data checking device, the electronic device and the readable storage medium provided in the embodiments of the present application acquire multiple sets of service data from multiple different data sources, respectively, where each set of service data is data corresponding to different associated service items in the same service scene, then process each set of service data to obtain a corresponding checking data set, and perform checking processing on data in any two checking data sets to detect whether the data in any two checking data sets meets a preset service consistency requirement. Therefore, the mode of checking and matching the processed core to the data set can be adopted, so that the service consistency detection among different service items under the same service scene can be realized, on one hand, the processing efficiency can be improved, the checking standard can be unified, and on the other hand, the consistency and the accuracy of the data can be ensured.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus and method embodiments described above are illustrative only, as the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, an electronic device, or a network device) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for data collation, the method comprising:
acquiring a plurality of groups of service data from a plurality of different data sources respectively, wherein each group of service data is data corresponding to different associated service items in the same service scene;
processing each group of service data to obtain a corresponding check data set;
and aiming at any two check data sets, checking the data in the two check data sets to detect whether the data in the two check data sets meet the preset service consistency requirement.
2. The data matching method as claimed in claim 1, wherein said step of matching the data in the two matching data sets to detect whether the data in the two matching data sets meets a predetermined service consistency requirement includes:
checking whether the data of the set fields in the two checking data sets are consistent or not and whether the arrangement sequence of the data of the set fields is consistent or not;
and if the data of the set fields are consistent and the arrangement sequence is consistent, determining that the data in the two check data sets meet the preset service consistency requirement.
3. The data matching method of claim 2, wherein said matching of said two sets of matching data to determine whether the data in said setting fields are identical and the data in said setting fields are arranged in the same order comprises:
aiming at one of the two verification data sets, obtaining data under a first set field in the verification data set;
obtaining a second set field corresponding to the first set field in the other of the two verification data sets, and obtaining data under the second set field in the verification data set;
and checking whether the data in the first set field is consistent with the data in the second set field and whether the data arrangement sequence is consistent.
4. The data collation method according to claim 1, wherein each group of service data is acquired based on a service logic module which is written in advance, and each group of service logic module contains an SQL statement;
the step of processing each group of service data to obtain a corresponding check data set includes:
syntax analysis is carried out on each group of service logic modules so as to detect whether the syntax of each group of service logic modules is correct or not;
and if the grammar of each group of business logic modules is correct, obtaining a corresponding check data set according to the SQL sentences in each group of business logic modules and the corresponding business data.
5. The data collation method according to claim 4, wherein said step of obtaining a corresponding collation data set based on the SQL statement and the corresponding business data in each group of business logic modules includes:
obtaining a service data set according to SQL sentences in each group of service logic modules and corresponding service data;
and when the expansion requirement of the data of the specific field in the service data set is obtained, acquiring the expansion data corresponding to the data under the specific field, and adding the expansion data into the service data set to obtain the check data set.
6. The data checking method according to claim 5, wherein the step of obtaining the service data set according to the SQL statements and the corresponding service data in each group of service logic modules comprises:
analyzing SQL sentences in each group of business logic modules by adopting a Durid middleware to obtain an analysis result;
and combining the analysis result and the acquired service data to obtain a corresponding service data set.
7. A data collation method according to any one of claims 1 to 6, wherein the method further comprises:
when the data in any two check data sets are determined not to meet the preset service consistency requirement, generating corresponding prompt information;
and sending the prompt information to a user subscribing the check event.
8. A data collating apparatus, characterized in that said apparatus comprises:
the acquisition module is used for respectively acquiring a plurality of groups of service data from a plurality of different data sources, wherein each group of service data is data corresponding to different associated service items in the same service scene;
the processing module is used for processing each group of service data to obtain a corresponding check data set;
and the checking module is used for checking the data in any two checking data sets so as to detect whether the data in the two checking data sets meet the preset service consistency requirement.
9. An electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing processor-executable machine-executable instructions that, when executed by the electronic device, are executed by the processors to perform the method steps of any of claims 1-7.
10. A computer-readable storage medium, characterized in that it stores machine-executable instructions which, when executed, implement the method steps of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110312665.6A CN112860722A (en) | 2021-03-24 | 2021-03-24 | Data checking method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110312665.6A CN112860722A (en) | 2021-03-24 | 2021-03-24 | Data checking method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112860722A true CN112860722A (en) | 2021-05-28 |
Family
ID=75992568
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110312665.6A Pending CN112860722A (en) | 2021-03-24 | 2021-03-24 | Data checking method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112860722A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168585A (en) * | 2021-12-16 | 2022-03-11 | 平安证券股份有限公司 | Data checking method, device, equipment and storage medium |
| CN116611793A (en) * | 2023-06-14 | 2023-08-18 | 中国长江三峡集团有限公司 | Service data induction method and system based on feature analysis |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106326219A (en) * | 2015-06-16 | 2017-01-11 | 阿里巴巴集团控股有限公司 | Business system data check method, apparatus and system |
| CN107291961A (en) * | 2017-08-08 | 2017-10-24 | 中国银行股份有限公司 | A kind of data processing method and device |
| CN109299222A (en) * | 2018-09-29 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Verification of data method and device |
-
2021
- 2021-03-24 CN CN202110312665.6A patent/CN112860722A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106326219A (en) * | 2015-06-16 | 2017-01-11 | 阿里巴巴集团控股有限公司 | Business system data check method, apparatus and system |
| CN107291961A (en) * | 2017-08-08 | 2017-10-24 | 中国银行股份有限公司 | A kind of data processing method and device |
| CN109299222A (en) * | 2018-09-29 | 2019-02-01 | 阿里巴巴集团控股有限公司 | Verification of data method and device |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168585A (en) * | 2021-12-16 | 2022-03-11 | 平安证券股份有限公司 | Data checking method, device, equipment and storage medium |
| CN116611793A (en) * | 2023-06-14 | 2023-08-18 | 中国长江三峡集团有限公司 | Service data induction method and system based on feature analysis |
| CN116611793B (en) * | 2023-06-14 | 2024-04-16 | 中国长江三峡集团有限公司 | Service data induction method and system based on feature analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US7177822B2 (en) | Common database system for sales and marketing process | |
| CN112882699B (en) | Service processing method, device, equipment and medium based on flow configuration engine | |
| CN109740122B (en) | Method and device for converting thinking guide graph use case file | |
| CN113886584A (en) | Information detection method, device and equipment for application program | |
| CN112416778B (en) | Test case recommendation method and device and electronic equipment | |
| CN109635300B (en) | Data verification method and device | |
| CN111327514B (en) | WeChat image-text message group sending method, system, server and storage medium | |
| CN112559865A (en) | Information processing system, computer-readable storage medium, and electronic device | |
| CN110427375B (en) | Method and device for identifying field type | |
| CN112860722A (en) | Data checking method and device, electronic equipment and readable storage medium | |
| CN114116496A (en) | Automatic testing method, device, equipment and medium | |
| CN112182391A (en) | User portrait drawing method and device | |
| CN112925664A (en) | Target user determination method and device, electronic equipment and storage medium | |
| CN111026753A (en) | Label storage method and device based on tree structure | |
| CN112732763A (en) | Data aggregation method and device, electronic equipment and medium | |
| CN113779092A (en) | Real-time data display method, device, equipment and medium based on data warehouse | |
| CN110457332B (en) | Information processing method and related equipment | |
| CN115328759A (en) | Form verification method and device | |
| CN113900955A (en) | Automatic testing method, device, equipment and storage medium | |
| CN117421214A (en) | Batch counting method, device, electronic equipment and computer readable storage medium | |
| CN109144999B (en) | Data positioning method, device, storage medium and program product | |
| CN111581937A (en) | Document generation method and device, computer readable medium and electronic equipment | |
| CN112445461B (en) | Business rule generation method and device, electronic equipment and readable storage medium | |
| US9471569B1 (en) | Integrating information sources to create context-specific documents | |
| CN112732681A (en) | Data platform migration method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210528 |