CN113886483A - Data verification method and device - Google Patents
Data verification method and device Download PDFInfo
- Publication number
- CN113886483A CN113886483A CN202111488866.8A CN202111488866A CN113886483A CN 113886483 A CN113886483 A CN 113886483A CN 202111488866 A CN202111488866 A CN 202111488866A CN 113886483 A CN113886483 A CN 113886483A
- Authority
- CN
- China
- Prior art keywords
- verification
- rules
- target data
- rule
- check
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/177—Editing, e.g. inserting or deleting of tables; using ruled lines
- G06F40/18—Editing, e.g. inserting or deleting of tables; using ruled lines of spreadsheets
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/166—Editing, e.g. inserting or deleting
- G06F40/186—Templates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/226—Validation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a data verification method and a device, wherein the method comprises the following steps: extracting target data from a database; configuring a plurality of check rules for the target data; the Spark memory calculation engine is used for verifying the target data according to the plurality of verification rules to obtain a verification result, so that the problems that in the related technology, data verification processing time is long, the verification rule updating is realized through production, and the requirement of immediate distribution and use cannot be met can be solved.
Description
Technical Field
The invention relates to the field of data processing, in particular to a data verification method and device.
Background
The monitoring organization effectively utilizes the financial institution submission data for analysis, has higher and higher requirements on the submission data, and clearly provides a large amount of data verification rules which are not regularly adjusted or newly added. At present, data verification processing time is long, verification rule updating is achieved through production, the requirement of immediate distribution and use cannot be met, and meanwhile, detailed data which cannot pass verification is large in size and cannot pass front-end query analysis.
Aiming at the problems that the data verification processing time is long, the verification rule updating is realized through production in the related technology, and the requirement of immediate distribution and use cannot be met, a solution is not provided.
Disclosure of Invention
The embodiment of the invention provides a data verification method and a data verification device, which at least solve the problems that the data verification processing time is long, the verification rule updating is realized through production, and the requirements of immediate distribution and immediate use cannot be met.
According to an embodiment of the present invention, there is provided a data verification method including:
extracting target data from a database;
configuring a plurality of check rules for the target data;
and verifying the target data according to the plurality of verification rules by using a spark memory calculation engine to obtain a verification result.
Optionally, configuring a plurality of verification rules for the target data includes:
acquiring a verification rule corresponding to the attribute of the target data from an ES server, and adjusting rule parameters of the verification rule corresponding to the attribute through an excel template to generate the multiple verification rules of the target data;
selecting a plurality of rule templates corresponding to a plurality of attributes for the target data through a client page, and configuring a plurality of check rules for the target data through the plurality of rule templates, wherein one attribute corresponds to one rule template.
Optionally, adjusting rule parameters of the check rule corresponding to the attribute through an excel template to generate the plurality of check rules of the target data includes:
acquiring an adjusting instruction which is input through touch operation and is used for adjusting rule parameters of the verification rule corresponding to the attribute in the excel template;
adjusting rule parameters of the check rules corresponding to the attributes according to the adjustment instruction;
and generating the plurality of verification rules of the target data according to the verification rules corresponding to the attributes and the corresponding adjusted rule parameters.
Optionally, configuring the plurality of verification rules for the target data through the plurality of rule templates includes:
respectively acquiring rule parameters input through touch operation in each rule template of the plurality of rule templates;
and generating the plurality of check rules for the target data based on the plurality of rule templates and the corresponding rule parameters.
Optionally, after configuring a plurality of verification rules for the target data, the method further includes:
associating the target data with the plurality of verification rules;
and uploading the associated target data and the plurality of check rules to the ES server.
Optionally, the verifying the target data according to the plurality of verification rules by using a spark memory calculation engine, and obtaining a verification result includes:
in the spark memory calculation engine, triggering a single spark memory calculation module for each of the plurality of check rules, and respectively checking the plurality of check rules based on the spark memory calculation module;
and releasing the computing resources of the spark memory computing module after the verification is completed.
Optionally, before triggering, in the spark memory computation engine, a single spark memory computation module for each of the plurality of check rules, the method further includes:
acquiring the total computing resource of the spark memory computing engine;
distributing the spark memory calculation modules for the plurality of check rules according to the total calculation resources, and setting check sequence numbers for the plurality of check rules, wherein the check sequence numbers are used for indicating a check sequence.
Optionally, after verifying the target data according to the plurality of verification rules by using a spark memory calculation engine to obtain a verification result, the method further includes:
loading the plurality of verification rules and the corresponding verification results into an ES database to provide detailed inquiry of the verification results of the target data, wherein the verification results comprise verified summary data and detail data which is not verified.
According to another embodiment of the present invention, there is also provided a data verification apparatus including:
the extraction module is used for extracting target data from the database;
the configuration module is used for configuring a plurality of check rules for the target data;
and the checking module is used for checking the target data according to the plurality of checking rules by using a spark memory calculation engine to obtain a checking result.
Optionally, the configuration module includes:
the adjusting submodule is used for acquiring the verification rule corresponding to the attribute of the target data from the ES server, and adjusting the rule parameters of the verification rule corresponding to the attribute through an excel template so as to generate the plurality of verification rules of the target data;
and the configuration submodule is used for selecting a plurality of rule templates corresponding to a plurality of attributes for the target data through a client page, and configuring a plurality of check rules for the target data through the plurality of rule templates, wherein one attribute corresponds to one rule template.
Optionally, the adjusting submodule is further used for
Acquiring an adjusting instruction which is input through touch operation and is used for adjusting rule parameters of the verification rule corresponding to the attribute in the excel template;
adjusting rule parameters of the check rules corresponding to the attributes according to the adjustment instruction;
and generating the plurality of verification rules of the target data according to the verification rules corresponding to the attributes and the corresponding adjusted rule parameters.
Optionally, the configuration submodule is further used for
Respectively acquiring rule parameters input through touch operation in each rule template of the plurality of rule templates;
and generating the plurality of check rules for the target data based on the plurality of rule templates and the corresponding rule parameters.
Optionally, the apparatus further comprises:
an association module for associating the target data with the plurality of verification rules;
and the uploading module is used for uploading the associated target data and the plurality of check rules to the ES server.
Optionally, the verification module includes:
the checking submodule is used for triggering a single spark memory calculation module for each checking rule in the spark memory calculation engine, and checking the plurality of checking rules based on the spark memory calculation modules respectively;
and the releasing submodule is used for releasing the computing resources of the spark memory computing module after the verification is finished.
Optionally, the apparatus further comprises:
the acquisition module is used for acquiring the total calculation resources of the spark memory calculation engine;
and the distribution module is used for distributing the spark memory calculation module for the plurality of check rules according to the total calculation resources and setting check serial numbers for the plurality of check rules, wherein the check serial numbers are used for indicating the check sequence.
Optionally, the apparatus further comprises:
the loading module is configured to load the plurality of verification rules and the corresponding verification results into an ES database to provide a detail query of the verification result of the target data, where the verification result includes summarized verified data and detail data that fails to be verified.
According to a further embodiment of the present invention, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps of any of the above-described method embodiments when executed.
According to yet another embodiment of the present invention, there is also provided an electronic device, including a memory in which a computer program is stored and a processor configured to execute the computer program to perform the steps in any of the above method embodiments.
By the method, the target data are extracted from the database; configuring a plurality of check rules for the target data; the Spark memory calculation engine is used for verifying the target data according to the plurality of verification rules to obtain a verification result, so that the problems that in the related technology, data verification processing time is long, verification rule updating is realized through production, and the requirement of immediate distribution and use cannot be met can be solved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:
fig. 1 is a block diagram of a hardware structure of a mobile terminal of a data verification method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of data verification according to an embodiment of the present invention;
FIG. 3 is a flow chart of configuration verification rules and data verification according to an embodiment of the present invention;
fig. 4 is a block diagram of a data verification apparatus according to an embodiment of the present invention.
Detailed Description
The invention will be described in detail hereinafter with reference to the accompanying drawings in conjunction with embodiments. It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.
Example 1
The method provided by the first embodiment of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. Taking a mobile terminal as an example, fig. 1 is a hardware structure block diagram of the mobile terminal of the data verification method according to the embodiment of the present invention, as shown in fig. 1, the mobile terminal may include one or more processors 102 (only one is shown in fig. 1) (the processor 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), and a memory 104 for storing data, and optionally, the mobile terminal may further include a transmission device 106 for a communication function and an input/output device 108. It will be understood by those skilled in the art that the structure shown in fig. 1 is only an illustration, and does not limit the structure of the mobile terminal. For example, the mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1.
The memory 104 may be used to store computer programs, for example, software programs and modules of application software, such as computer programs corresponding to the data verification method in the embodiment of the present invention, and the processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, so as to implement the above-mentioned method. The memory 104 may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory located remotely from the processor 102, which may be connected to the mobile terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The transmission device 106 is used to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a Network adapter (NIC) that can be connected to other Network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.
In this embodiment, a data verification method operating in the mobile terminal or the network architecture is provided, and fig. 2 is a flowchart of the data verification method according to the embodiment of the present invention, as shown in fig. 2, the flowchart includes the following steps:
step S202, extracting target data from a database;
step S204, configuring a plurality of check rules for the target data;
and step S206, verifying the target data according to the plurality of verification rules by using a spark memory calculation engine to obtain a verification result.
Extracting target data from the database through the above steps S202 to S206; configuring a plurality of check rules for the target data; the Spark memory calculation engine is used for verifying the target data according to the plurality of verification rules to obtain a verification result, so that the problems that in the related technology, data verification processing time is long, verification rule updating is realized through production, and the requirement of immediate distribution and use cannot be met can be solved.
In this embodiment of the present invention, the step S204 may specifically include:
s2041, obtaining a verification rule corresponding to an attribute of the target data from an ES server, adjusting a rule parameter of the verification rule corresponding to the attribute through an excel template to generate the multiple verification rules of the target data, and further obtaining an adjustment instruction input through a touch operation to adjust the rule parameter of the verification rule corresponding to the attribute in the excel template;
adjusting rule parameters of the check rules corresponding to the attributes according to the adjustment instruction; and generating the plurality of verification rules of the target data according to the verification rules corresponding to the attributes and the corresponding adjusted rule parameters.
S2042, selecting a plurality of rule templates corresponding to a plurality of attributes for the target data through a client page, and configuring the plurality of verification rules for the target data through the plurality of rule templates, where one attribute corresponds to one rule template, and further, obtaining rule parameters input through touch operation in each rule template of the plurality of rule templates, respectively; and generating the plurality of check rules for the target data based on the plurality of rule templates and the corresponding rule parameters.
In an optional embodiment, after the step S204, the target data is associated with the plurality of check rules; and uploading the associated target data and the plurality of check rules to the ES server.
In this embodiment of the present invention, the step S206 may specifically include:
s2061, in the spark memory calculation engine, triggering a single spark memory calculation module for each of the plurality of check rules, and checking the plurality of check rules based on the spark memory calculation modules, respectively;
s2062, releasing the computing resources of the spark memory computing module after the verification is completed.
In an optional embodiment, before the step S2061, the total computing resources of the spark memory computing engine are obtained; distributing the spark memory calculation modules for the plurality of check rules according to the total calculation resources, and setting check sequence numbers for the plurality of check rules, wherein the check sequence numbers are used for indicating a check sequence.
In another optional embodiment, after the step S206, the plurality of verification rules and the corresponding verification results are loaded into an ES database to provide a detail query of the verification result of the target data, where the verification result includes the summarized verified data and the detail data that is not verified.
In the embodiment of the invention, a data verification system adopts a spark memory calculation model and a spark client to concurrently receive a user submission request, encapsulates the memory of an executor specified by a user, the size of a CPU and the number of the exectors, submits the memory and the CPU to a yarn cluster resource queue, and allocates each task exclusive resource for verification processing. Each task is used as a table for checking, each table can have a plurality of rules, main table data is read only once and loaded into a storage memory, and the main table data in the storage memory is used for checking regardless of the number of checking rules in the follow-up process. The data do not need to be read repeatedly, the data movement is reduced, the disk IO is reduced, the network IO is reduced, and the memory calculation speed is high. The method comprises the steps that an execution memory is used in the program or sql execution process, the execution memory is large enough, all process processing is calculated in the memory, spark and ES have good compatibility, check result data and error detail data generated after check are directly updated to an ES (elastic search) index database, and whether the data exist, namely update and are not inserted is automatically detected. After the verification task is executed, resources are released immediately and are provided for other tasks to use, and a plurality of spark tasks automatically allocate the resources to use according to the parameters specified by the user. And if the resources are insufficient, the task enters the queue to wait until the resources are used, and the application of resource execution is carried out.
The embodiment of the invention utilizes the combination of the flexible allocation advantage of Spark memory computing resources and the reverse index of the ES search engine, solves the problems of instant allocation and use of the check rule and the front-end query analysis support of the large data volume check result, and realizes the configuration and execution of the check rule diversified in different scenes of the user. Fig. 3 is a flowchart of configuration verification rule and data verification according to an embodiment of the present invention, as shown in fig. 3, including:
step S301, configuring a verification rule for data in a verification system, specifically, configuring a plurality of verification rules for one data;
step S302, uploading a verification result generated in the data verification process and detail data which cannot pass the verification to an ES database;
step S303, the ES database is used for storing the verification rule, the verification result and the detail data which cannot pass the verification in a correlation mode.
Aiming at different scenes and requirements, two modes are provided for the configuration of the verification rule, and the rule can be turned on and off as required. The first mode is that a user maintains rules in an excel template and guides a configured template into an ES server for a verification engine to use, and the mode is more suitable for fixed large-batch verification rule setting; the second mode is front-end page configuration, the operation of the mode is simplified, the general branch can flexibly define the rule, and the method also supports partial simple SQL to compile the customized rule, and the mode is more suitable for the improvement of a small amount of temporary rules.
The data verification utilizes the advantages of few IO read-write and high parallelism of spark memory calculation engines, each verification rule triggers a single spark calculation module, resources of each calculation module are shared independently, spark calculation resources are automatically released after verification is completed, and a plurality of rules can automatically queue and distribute according to spark calculation total resources.
The verification result in the embodiment of the invention comprises verified summarized data and detail data which are not passed, and the data are directly loaded to the ES database by utilizing the good compatibility of the spark memory calculation engine and the ES database, so that the user can inquire and analyze from the general situation to the specific details.
The functions are commercialized, business personnel and technical personnel can understand the functions more conveniently, the functions are separated from the technical bottom layer, the learning cost is low, and the rule entry is more convenient to understand. The method supports simple sql and page operation rule input, rules can be flexibly configured, rule isolation is supported, the general branch can flexibly configure own rules, rules of others can be used, the rules can be divided into batches, and different rules can be customized in different months.
The data is analyzed by the memory calculation engine based on big data Spark. And a memory mode is utilized for data calculation, data landing is reduced, and disk IO is reduced. Spark is good in compatibility with hive and es, Spark supports hivesql, and can operate es data in batches.
Example 2
According to another embodiment of the present invention, there is also provided a data verification apparatus, and fig. 4 is a block diagram of the data verification apparatus according to the embodiment of the present invention, as shown in fig. 4, including:
an extraction module 42 for extracting target data from the database;
a configuration module 44, configured to configure a plurality of check rules for the target data;
and the checking module 46 is configured to check the target data according to the plurality of checking rules by using a spark memory calculation engine to obtain a checking result.
Optionally, the configuration module 44 includes:
the adjusting submodule is used for acquiring the verification rule corresponding to the attribute of the target data from the ES server, and adjusting the rule parameters of the verification rule corresponding to the attribute through an excel template so as to generate the plurality of verification rules of the target data;
and the configuration submodule is used for selecting a plurality of rule templates corresponding to a plurality of attributes for the target data through a client page, and configuring a plurality of check rules for the target data through the plurality of rule templates, wherein one attribute corresponds to one rule template.
Optionally, the adjusting submodule is further used for
Acquiring an adjusting instruction which is input through touch operation and is used for adjusting rule parameters of the verification rule corresponding to the attribute in the excel template;
adjusting rule parameters of the check rules corresponding to the attributes according to the adjustment instruction;
and generating the plurality of verification rules of the target data according to the verification rules corresponding to the attributes and the corresponding adjusted rule parameters.
Optionally, the configuration submodule is further used for
Respectively acquiring rule parameters input through touch operation in each rule template of the plurality of rule templates;
and generating the plurality of check rules for the target data based on the plurality of rule templates and the corresponding rule parameters.
Optionally, the apparatus further comprises:
an association module for associating the target data with the plurality of verification rules;
and the uploading module is used for uploading the associated target data and the plurality of check rules to the ES server.
Optionally, the verification module includes:
the checking submodule is used for triggering a single spark memory calculation module for each checking rule in the spark memory calculation engine, and checking the plurality of checking rules based on the spark memory calculation modules respectively;
and the releasing submodule is used for releasing the computing resources of the spark memory computing module after the verification is finished.
Optionally, the apparatus further comprises:
the acquisition module is used for acquiring the total calculation resources of the spark memory calculation engine;
and the distribution module is used for distributing the spark memory calculation module for the plurality of check rules according to the total calculation resources and setting check serial numbers for the plurality of check rules, wherein the check serial numbers are used for indicating the check sequence.
Optionally, the apparatus further comprises:
the loading module is configured to load the plurality of verification rules and the corresponding verification results into an ES database to provide a detail query of the verification result of the target data, where the verification result includes summarized verified data and detail data that fails to be verified.
It should be noted that, the above modules may be implemented by software or hardware, and for the latter, the following may be implemented, but not limited to: the modules are all positioned in the same processor; alternatively, the modules are respectively located in different processors in any combination.
Example 3
Embodiments of the present invention also provide a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the storage medium may be configured to store a computer program for executing the steps of:
s1, extracting target data from the database;
s2, configuring a plurality of check rules for the target data;
and S3, verifying the target data according to the plurality of verification rules by using a spark memory calculation engine to obtain a verification result.
Optionally, in this embodiment, the storage medium may include, but is not limited to: various media capable of storing computer programs, such as a usb disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
Example 4
Embodiments of the present invention also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the above method embodiments.
Optionally, the electronic apparatus may further include a transmission device and an input/output device, wherein the transmission device is connected to the processor, and the input/output device is connected to the processor.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, extracting target data from the database;
s2, configuring a plurality of check rules for the target data;
and S3, verifying the target data according to the plurality of verification rules by using a spark memory calculation engine to obtain a verification result.
Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation manners, and this embodiment is not described herein again.
It will be apparent to those skilled in the art that the modules or steps of the present invention described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (11)
1. A method for data verification, comprising:
extracting target data from a database;
configuring a plurality of check rules for the target data;
and verifying the target data according to the plurality of verification rules by using a spark memory calculation engine to obtain a verification result.
2. The method of claim 1, wherein configuring the target data with a plurality of validation rules comprises:
acquiring a verification rule corresponding to the attribute of the target data from an ES server, and adjusting rule parameters of the verification rule corresponding to the attribute through an excel template to generate the multiple verification rules of the target data;
selecting a plurality of rule templates corresponding to a plurality of attributes for the target data through a client page, and configuring a plurality of check rules for the target data through the plurality of rule templates, wherein one attribute corresponds to one rule template.
3. The method according to claim 2, wherein adjusting rule parameters of the check rule corresponding to the attribute through an excel template to generate the plurality of check rules of the target data comprises:
acquiring an adjusting instruction which is input through touch operation and is used for adjusting rule parameters of the verification rule corresponding to the attribute in the excel template;
adjusting rule parameters of the check rules corresponding to the attributes according to the adjustment instruction;
and generating the plurality of verification rules of the target data according to the verification rules corresponding to the attributes and the corresponding adjusted rule parameters.
4. The method of claim 2, wherein configuring the plurality of validation rules for the target data via the plurality of rule templates comprises:
respectively acquiring rule parameters input through touch operation in each rule template of the plurality of rule templates;
and generating the plurality of check rules for the target data based on the plurality of rule templates and the corresponding rule parameters.
5. The method of claim 2, wherein after configuring the plurality of verification rules for the target data, the method further comprises:
associating the target data with the plurality of verification rules;
and uploading the associated target data and the plurality of check rules to the ES server.
6. The method according to claim 1, wherein the verifying the target data according to the plurality of verification rules by using a spark memory calculation engine comprises:
in the spark memory calculation engine, triggering a single spark memory calculation module for each of the plurality of check rules, and respectively checking the plurality of check rules based on the spark memory calculation module;
and releasing the computing resources of the spark memory computing module after the verification is completed.
7. The method of claim 6, wherein before triggering, in the spark memory computation engine, a single spark memory computation module for each of the plurality of verification rules, the method further comprises:
acquiring the total computing resource of the spark memory computing engine;
distributing the spark memory calculation modules for the plurality of check rules according to the total calculation resources, and setting check sequence numbers for the plurality of check rules, wherein the check sequence numbers are used for indicating a check sequence.
8. The method according to any one of claims 1 to 7, wherein after verifying the target data according to the plurality of verification rules by using a spark memory calculation engine, the method further comprises:
loading the plurality of verification rules and the corresponding verification results into an ES database to provide detailed inquiry of the verification results of the target data, wherein the verification results comprise verified summary data and detail data which is not verified.
9. A data verification apparatus, comprising:
the extraction module is used for extracting target data from the database;
the configuration module is used for configuring a plurality of check rules for the target data;
and the checking module is used for checking the target data according to the plurality of checking rules by using a spark memory calculation engine to obtain a checking result.
10. A computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to carry out the method of any one of claims 1 to 8 when executed.
11. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, and wherein the processor is arranged to execute the computer program to perform the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111488866.8A CN113886483A (en) | 2021-12-08 | 2021-12-08 | Data verification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111488866.8A CN113886483A (en) | 2021-12-08 | 2021-12-08 | Data verification method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113886483A true CN113886483A (en) | 2022-01-04 |
Family
ID=79016537
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111488866.8A Pending CN113886483A (en) | 2021-12-08 | 2021-12-08 | Data verification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113886483A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190155801A1 (en) * | 2017-08-16 | 2019-05-23 | Walmart Apollo, Llc | Systems and methods for distributed data validation |
| CN112199416A (en) * | 2020-09-30 | 2021-01-08 | 支付宝(杭州)信息技术有限公司 | Data rule generation method and device |
| CN112396419A (en) * | 2020-12-08 | 2021-02-23 | 深圳前海微众银行股份有限公司 | Method, device and equipment for generating check rule and storage medium |
| CN113742776A (en) * | 2021-09-08 | 2021-12-03 | 未鲲(上海)科技服务有限公司 | Data verification method and device based on biological recognition technology and computer equipment |
-
2021
- 2021-12-08 CN CN202111488866.8A patent/CN113886483A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190155801A1 (en) * | 2017-08-16 | 2019-05-23 | Walmart Apollo, Llc | Systems and methods for distributed data validation |
| CN112199416A (en) * | 2020-09-30 | 2021-01-08 | 支付宝(杭州)信息技术有限公司 | Data rule generation method and device |
| CN112396419A (en) * | 2020-12-08 | 2021-02-23 | 深圳前海微众银行股份有限公司 | Method, device and equipment for generating check rule and storage medium |
| CN113742776A (en) * | 2021-09-08 | 2021-12-03 | 未鲲(上海)科技服务有限公司 | Data verification method and device based on biological recognition technology and computer equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11386058B2 (en) | Rule-based autonomous database cloud service framework | |
| US10496532B1 (en) | Automatically reconfiguring a performance test environment | |
| US20190102384A1 (en) | Autonomous multitenant database cloud service framework | |
| CN109995677B (en) | Resource allocation method, device and storage medium | |
| CN106778351B (en) | Data desensitization method and device | |
| US20140289286A1 (en) | System and method for performance tuning of garbage collection algorithms | |
| JP5730386B2 (en) | Computer system and parallel distributed processing method | |
| EP3605364B1 (en) | Query processing method, data source registration method and query engine | |
| CN112800095A (en) | Data processing method, device, equipment and storage medium | |
| CN108197091B (en) | Method, system and related equipment for creating data table | |
| CN114691786A (en) | Method and device for determining data blood relationship, storage medium and electronic device | |
| US10102098B2 (en) | Method and system for recommending application parameter setting and system specification setting in distributed computation | |
| CN110569129A (en) | Resource allocation method and device, storage medium and electronic device | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN112182031B (en) | Data query method and device, storage medium and electronic device | |
| CN110990381B (en) | Processing method and device of server, storage medium and electronic device | |
| KR20200135522A (en) | Transaction processing method, server and transaction processing system | |
| CN112163948A (en) | Method, system, equipment and storage medium for separately-moistening calculation | |
| CN113886483A (en) | Data verification method and device | |
| CN115048359A (en) | Universal data processing architecture design method for multi-type access data sources | |
| CN113110982B (en) | Data access layer verification method and device, storage medium and electronic device | |
| CN110362575B (en) | Method and device for generating global index of data | |
| CN111880864B (en) | HTTP-based model calling method, system, computer device and storage medium | |
| CN114564249A (en) | Recommendation scheduling engine, recommendation scheduling method, and computer-readable storage medium | |
| CN113268507A (en) | Database data reading system, method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220104 |