CN110309125B - Data verification method, electronic device and storage medium - Google Patents
Data verification method, electronic device and storage medium Download PDFInfo
- Publication number
- CN110309125B CN110309125B CN201910559406.6A CN201910559406A CN110309125B CN 110309125 B CN110309125 B CN 110309125B CN 201910559406 A CN201910559406 A CN 201910559406A CN 110309125 B CN110309125 B CN 110309125B
- Authority
- CN
- China
- Prior art keywords
- data
- index
- service data
- configuration
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/217—Database tuning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of data processing, and provides a data checking method, an electronic device and a computer storage medium, wherein the method comprises the following steps: the method comprises the steps that through connection of service systems corresponding to a plurality of data sources, corresponding service data are obtained from each service system according to preset time frequency to carry out index configuration, indexes and index configuration information corresponding to the service data are obtained, and the indexes corresponding to the service data are subjected to regular configuration to generate configuration files comprising index threshold values; inputting the indexes corresponding to the business data and the index configuration information into a calculation engine for operation to obtain the index values corresponding to the business data; and finally, reading the configuration file by using a rule engine to analyze the index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value. The invention ensures the reliability of the data by regularly carrying out automatic verification after the data of each service system is regularly configured, thereby improving the data quality.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data verification method, an electronic device, and a computer-readable storage medium.
Background
With the popularization of big data technology, data processing and calculation have wide application in various industry fields. Among them, the ETL (Extract-Transform-Load) technique plays an important role in large-scale data processing.
At present, service data of a plurality of companies are reported and summarized through service systems in various places, ETL has high requirements on data quality in application scenes such as data reporting, index statistics, decision analysis and the like, and a traditional data verification mode adopts partial data manual logarithm, so that the defects of low verification efficiency, easy error of manual logarithm, low automation degree and the like exist, the accuracy of data is influenced, and the decision of the data is wrong.
Disclosure of Invention
In view of the above, the present invention provides a data verification method, an electronic device, and a computer-readable storage medium, which are mainly intended to automatically verify data of each service system periodically, to ensure reliability of the data, and to improve data quality.
In order to achieve the above object, the present invention provides a data verification method applied to an electronic device, the method comprising:
an acquisition step: connecting the service systems corresponding to the data sources, and acquiring corresponding service data from each service system according to a preset time frequency;
a first configuration step: index configuration is carried out on each acquired service data to obtain indexes and index configuration information corresponding to each service data;
a second configuration step: carrying out rule configuration on indexes corresponding to the obtained business data to generate a configuration file, wherein the configuration file comprises index thresholds corresponding to the business data;
a calculation step: inputting the indexes and the index configuration information corresponding to the business data into a calculation engine for calculation to obtain the index values corresponding to the calculated business data; and
a checking step: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
Preferably, the verifying step comprises:
and when the index value corresponding to one piece of service data exceeds the corresponding index threshold value, judging that the service data is wrong, distributing a preset mark for the wrong service data and storing the mark to an early warning table.
Preferably, after the verifying step, the method further comprises:
early warning step: and scanning the early warning table at regular time, and triggering a preset early warning system to generate corresponding early warning information and sending the early warning information to a corresponding service system when the preset marked service data is scanned.
Preferably, the index configuration information includes a data source associated with the index, an operation logic of the index, and a configuration time.
Preferably, the calculating step comprises:
finding out the operation logic of the index corresponding to each service data from the obtained index configuration information;
and carrying out SQL calculation on the indexes corresponding to the business data according to the found operation logic to obtain the index values corresponding to the business data.
In addition, to achieve the above object, the present invention further provides an electronic device, which includes a memory and a processor, wherein the memory stores a data verification program operable on the processor, and the data verification program, when executed by the processor, implements the following steps:
an acquisition step: connecting the service systems corresponding to the data sources, and acquiring corresponding service data from each service system according to a preset time frequency;
a first configuration step: index configuration is carried out on each acquired service data to obtain indexes and index configuration information corresponding to each service data;
a second configuration step: carrying out rule configuration on indexes corresponding to the obtained business data to generate a configuration file, wherein the configuration file comprises index thresholds corresponding to the business data;
a calculation step: inputting the indexes and the index configuration information corresponding to the business data into a calculation engine for calculation to obtain the index values corresponding to the calculated business data; and
a checking step: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
Preferably, the verifying step comprises:
and when the index value corresponding to one piece of service data exceeds the corresponding index threshold value, judging that the service data is wrong, distributing a preset mark for the wrong service data and storing the mark to an early warning table.
Preferably, after the verifying step, the method further comprises:
early warning step: and scanning the early warning table at regular time, and triggering a preset early warning system to generate corresponding early warning information and sending the early warning information to a corresponding service system when the preset marked service data is scanned.
Preferably, the index configuration information includes a data source associated with the index, an operation logic of the index, and a configuration time.
In addition, to achieve the above object, the present invention further provides a computer-readable storage medium, where the computer-readable storage medium includes a data verification program, and when the data verification program is executed by a processor, any step of the data verification method described above can be implemented.
The data verification method, the electronic device and the computer readable storage medium provided by the invention have the advantages that through connecting the service systems corresponding to a plurality of data sources, corresponding service data are obtained from each service system according to the preset time frequency, index configuration is carried out on each obtained service data, indexes and index configuration information corresponding to each service data are obtained, the indexes corresponding to each obtained service data are regularly configured to generate a configuration file, and the configuration file comprises index thresholds corresponding to each service data; inputting the indexes and the index configuration information corresponding to the business data into a calculation engine for calculation to obtain the index values corresponding to the calculated business data; and finally, reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value. The invention ensures the reliability of the data by regularly carrying out automatic verification after the data of each service system is regularly configured, thereby improving the data quality.
Drawings
FIG. 1 is a diagram of an electronic device according to a preferred embodiment of the present invention;
FIG. 2 is a block diagram of a preferred embodiment of the data verification process of FIG. 1;
FIG. 3 is a flow chart of a data verification method according to a preferred embodiment of the present invention;
FIG. 4 is a block diagram of another preferred embodiment of the data verification process of FIG. 1;
FIG. 5 is a flow chart of another preferred embodiment of a data verification method according to the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the description relating to "first", "second", etc. in the present invention is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
Referring to fig. 1, a schematic diagram of an electronic device according to a preferred embodiment of the invention is shown. The electronic apparatus 1 is a device capable of automatically performing numerical calculation and/or information processing in accordance with a command set or stored in advance. The electronic device 1 may be a computer, or may be a single network server, a server group composed of a plurality of network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing, where cloud computing is one of distributed computing and is a super virtual computer composed of a group of loosely coupled computers.
In the present embodiment, the electronic device 1 may include, but is not limited to, a memory 11, a processor 12, and a network interface 13, which are communicatively connected to each other through a system bus, and the memory 11 stores a data verification program 10 that can be executed on the processor 12. It is noted that fig. 1 only shows the electronic device 1 with components 11-13, but it is to be understood that not all shown components are required to be implemented, and that more or less components may be implemented instead.
The storage 11 includes a memory and at least one type of readable storage medium. The memory provides cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic apparatus 1, such as a hard disk of the electronic apparatus 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic apparatus 1, such as a plug-in hard disk provided on the electronic apparatus 1, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. In this embodiment, the readable storage medium of the memory 11 is generally used for storing an operating system and various application software installed in the electronic device 1, for example, storing the data verification program 10 in an embodiment of the present invention. Further, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 12 may be a Central Processing Unit (CPU), controller, microcontroller, microprocessor, or other data Processing chip in some embodiments. The processor 12 is generally used for controlling the overall operation of the electronic apparatus 1, such as performing control and processing related to data interaction or communication with the other devices. In this embodiment, the processor 12 is configured to run the program code stored in the memory 11 or process data, for example, run the data verification program 10.
The network interface 13 may comprise a wireless network interface or a wired network interface, and the network interface 13 is generally used for establishing a communication connection between the electronic apparatus 1 and other electronic devices.
The data verification program 10 is stored in the memory 11 and includes computer readable instructions stored in the memory 11 that are executable by the processor 12 to implement the methods of the embodiments of the present application.
In an embodiment, the data verification program 10 when executed by the processor 12 implements the following steps:
an acquisition step: and connecting the service systems corresponding to the plurality of data sources, and acquiring corresponding service data from each service system according to a preset time frequency.
In this embodiment, the data source includes a service system name, an IP domain name, a connection port, a database type, a data synchronization frequency, and the like. The data source is connected to each service system, corresponding service data are obtained from each service system according to the preset time frequency, the obtained service data are solidified to the Hive warehouse, other data are continuously input into each service system, and the reliability of the obtained original service data is guaranteed. The Hive is a data warehouse tool, and can extract, convert and load the acquired data, and can also perform SQL data query and the like.
In an embodiment, service systems corresponding to a plurality of data sources (such as IP domain names or connection ports of different systems) are connected, and corresponding service data (policy data) is obtained from each service system (such as service systems of provinces and cities) according to a preset time frequency (such as 8 am every day) and is solidified to the Hive warehouse.
A first configuration step: and performing index configuration on each acquired service data to obtain indexes and index configuration information corresponding to each service data.
Specifically, index configuration is performed on the service data according to the field attribute of each acquired service data to obtain an index corresponding to each service data, the obtained indexes corresponding to each service data are correlated, the correlated indexes are configured and matched with a preset operation logic, and finally index configuration information is obtained.
Further, the index configuration information includes a data source associated with the index, an operation logic of the index, and a configuration time.
The data source related to the indexes comprises the association between indexes corresponding to all the service data and the association between the data sources of all the service data, so that the data sources can be traced conveniently;
the operation logic of the index comprises a general function (such as a function of sum, count, min or max) and a preset custom function (such as a function of a mathematical formula);
the configuration time refers to the time record of index configuration of each service data.
A second configuration step: and carrying out rule configuration on the indexes corresponding to the obtained service data to generate a configuration file, wherein the configuration file comprises the index threshold corresponding to the service data.
In this embodiment, in order to automatically verify the acquired service data, the service data is configured in a rule manner, that is, indexes corresponding to the service data are configured in advance, and a configuration file is generated, where the configuration file is a file in drl format and can be analyzed by a Drools rule engine, the configuration file includes an index threshold corresponding to each service data, and the index threshold is generated when the rule is configured according to the field attribute of each service data and the corresponding index.
A calculation step: and inputting the indexes corresponding to the business data and the index configuration information into a calculation engine for calculation to obtain the index values corresponding to the calculated business data.
In this embodiment, the computation engine is a Spark engine, which is a Spark engine for rapidly and generally processing large-scale data. And performing Spark engine calculation according to the indexes and the index configuration information corresponding to each service data, wherein the Spark engine calculation comprises calculation by using an SQL tool in the Spark engine and/or calculation between dimension levels of the service data of each service system by using the Spark engine, so as to obtain the index values corresponding to each service data after calculation, and storing the index values in a json format to a database.
SQL is an abbreviation for Structured Query Language (Structured Query Language). The method is a tool for data query, programming and data management, and has universality for processing large-scale data calculation and providing a plurality of different calculation modes.
Further, the calculating step includes:
finding out the operation logic of the index corresponding to each service data from the obtained index configuration information;
and carrying out SQL calculation on the indexes corresponding to the business data according to the found operation logic to obtain the index values corresponding to the business data.
The operation logic comprises a general function (such as a sum function, a count function, a min function or a max function) and a preset custom function (such as a function of a mathematical formula), and the SQL is used for completing complex and various service data calculations according to the operation logic configured by the indexes corresponding to the service data, so as to obtain the index values corresponding to the service data.
A checking step: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
In this embodiment, the rule engine adopts a Drools engine, which can automatically load and analyze a configuration file, and for the configuration file after the business data rule configuration is completed, the Drools engine automatically reads the configuration file and analyzes an index threshold corresponding to each business data to check whether an index value corresponding to each computed business data exceeds the corresponding index threshold, so that the index value corresponding to each computed business data can be quickly and efficiently checked, and the correctness of each business data can be determined.
Further, the verifying step includes:
and when the index value corresponding to one piece of service data exceeds the corresponding index threshold value, judging that the service data is wrong, distributing a preset mark for the wrong service data and storing the mark to an early warning table.
Further, the verifying step further comprises:
early warning step: and scanning the early warning table at regular time, and triggering a preset early warning system to generate corresponding early warning information and sending the early warning information to a corresponding service system when the preset marked service data is scanned.
In one embodiment, the early warning table is scanned regularly for a preset time, when the preset marked service data is scanned, a preset early warning system is triggered to generate corresponding early warning information, the early warning information comprises marked error service data, a data source related to the error service data, early warning time and the like, and the early warning information is sent to the corresponding service system. After the service system receives the early warning information, an operator of the service system analyzes according to the early warning information, and timely solves the problem of wrong service data, so that the correctness of the data is ensured, and the quality of the data is improved.
Referring to FIG. 2, a block diagram of a preferred embodiment of the data verification process 10 of FIG. 1 is shown.
In one embodiment, the data verification program 10 includes: the system comprises an acquisition module 101, a first configuration module 102, a second configuration module 103, a calculation module 104 and a verification module 105. The functions or operation steps implemented by the module 101-105 are similar to those of the following data verification method, and are not described in detail here, for example, where:
the acquisition module 101 is configured to connect to service systems corresponding to multiple data sources, and acquire corresponding service data from each service system according to a preset time frequency;
the first configuration module 102 is configured to perform index configuration on each acquired service data to obtain an index and index configuration information corresponding to each service data;
the second configuration module 103 is configured to perform rule configuration on the obtained indexes corresponding to each service data to generate a configuration file, where the configuration file includes an index threshold corresponding to each service data;
the calculation module 104 is configured to input the index and the index configuration information corresponding to each service data into a calculation engine for calculation, so as to obtain an index value corresponding to each calculated service data; and
the checking module 105 is configured to read the configuration file by using a rule engine, analyze the configuration file to obtain an index threshold corresponding to each service data, and check whether an index value corresponding to each service data exceeds the corresponding index threshold.
Referring to fig. 4, which is a block diagram of another preferred embodiment of the data verification process 10 of fig. 1, after the verification module 105, the data verification process 10 further includes an early warning module 106, which illustratively:
and the early warning module 106 is configured to scan the early warning table at regular time, trigger a preset early warning system to generate corresponding early warning information when the preset marked service data is scanned, and send the corresponding early warning information to the corresponding service system.
Referring to fig. 3, a flow chart of a data verification method according to a preferred embodiment of the invention is shown. The invention discloses a data verification method, which is applied to the electronic device and comprises the following steps:
step S210: and connecting the service systems corresponding to the plurality of data sources, and acquiring corresponding service data from each service system according to a preset time frequency.
In this embodiment, the data source includes a service system name, an IP domain name, a connection port, a database type, a data synchronization frequency, and the like. The data source is connected to each service system, corresponding service data are obtained from each service system according to the preset time frequency, the obtained service data are solidified to the Hive warehouse, other data are continuously input into each service system, and the reliability of the obtained original service data is guaranteed. The Hive is a data warehouse tool, and can extract, convert and load the acquired data, and can also perform SQL data query and the like.
In an embodiment, service systems corresponding to a plurality of data sources (such as IP domain names or connection ports of different systems) are connected, and corresponding service data (policy data) is obtained from each service system (such as service systems of provinces and cities) according to a preset time frequency (such as 8 am every day) and is solidified to the Hive warehouse.
Step S220: and performing index configuration on each acquired service data to obtain indexes and index configuration information corresponding to each service data.
Specifically, index configuration is performed on the service data according to the field attribute of each acquired service data to obtain an index corresponding to each service data, the obtained indexes corresponding to each service data are correlated, the correlated indexes are configured and matched with a preset operation logic, and finally index configuration information is obtained.
Further, the index configuration information includes a data source associated with the index, an operation logic of the index, and a configuration time.
The data source related to the indexes comprises the association between indexes corresponding to all the service data and the association between the data sources of all the service data, so that the data sources can be traced conveniently;
the operation logic of the index comprises a general function (such as a function of sum, count, min or max) and a preset custom function (such as a function of a mathematical formula);
the configuration time refers to the time record of index configuration of each service data.
Step S230: and carrying out rule configuration on the indexes corresponding to the obtained service data to generate a configuration file, wherein the configuration file comprises the index threshold corresponding to the service data.
In this embodiment, in order to automatically verify the acquired service data, the service data is configured in a rule manner, that is, indexes corresponding to the service data are configured in advance, and a configuration file is generated, where the configuration file is a file in drl format and can be analyzed by a Drools rule engine, the configuration file includes an index threshold corresponding to each service data, and the index threshold is generated when the rule is configured according to the field attribute of each service data and the corresponding index.
Step S240: and inputting the indexes corresponding to the business data and the index configuration information into a calculation engine for calculation to obtain the index values corresponding to the calculated business data.
In this embodiment, the computation engine is a Spark engine, which is a Spark engine for rapidly and generally processing large-scale data. And performing Spark engine calculation according to the indexes and the index configuration information corresponding to each service data, wherein the Spark engine calculation comprises calculation by using an SQL tool in the Spark engine and/or calculation between dimension levels of the service data of each service system by using the Spark engine, so as to obtain the index values corresponding to each service data after calculation, and storing the index values in a json format to a database.
SQL is an abbreviation for Structured Query Language (Structured Query Language). The method is a tool for data query, programming and data management, and has universality for processing large-scale data calculation and providing a plurality of different calculation modes.
Further, the step S240: the method comprises the following steps:
finding out the operation logic of the index corresponding to each service data from the obtained index configuration information;
and carrying out SQL calculation on the indexes corresponding to the business data according to the found operation logic to obtain the index values corresponding to the business data.
The operation logic comprises a general function (such as a sum function, a count function, a min function or a max function) and a preset custom function (such as a function of a mathematical formula), and the SQL is used for completing complex and various service data calculations according to the operation logic configured by the indexes corresponding to the service data, so as to obtain the index values corresponding to the service data.
Step S250: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
In this embodiment, the rule engine adopts a Drools engine, which can automatically load and analyze a configuration file, and for the configuration file after the business data rule configuration is completed, the Drools engine automatically reads the configuration file and analyzes an index threshold corresponding to each business data to check whether an index value corresponding to each computed business data exceeds the corresponding index threshold, so that the index value corresponding to each computed business data can be quickly and efficiently checked, and the correctness of each business data can be determined.
Further, the step S250: the method comprises the following steps:
and when the index value corresponding to one piece of service data exceeds the corresponding index threshold value, judging that the service data is wrong, distributing a preset mark for the wrong service data and storing the mark to an early warning table.
Referring to fig. 5, which is a flowchart illustrating another preferred embodiment of the data verification method according to the present invention, after step S250, the method further includes:
step S260: and scanning the early warning table at regular time, and triggering a preset early warning system to generate corresponding early warning information and sending the early warning information to a corresponding service system when the preset marked service data is scanned.
In one embodiment, the early warning table is scanned regularly for a preset time, when the preset marked service data is scanned, a preset early warning system is triggered to generate corresponding early warning information, the early warning information comprises marked error service data, a data source related to the error service data, early warning time and the like, and the early warning information is sent to the corresponding service system. After the service system receives the early warning information, an operator of the service system analyzes according to the early warning information, and timely solves the problem of wrong service data, so that the correctness of the data is ensured, and the quality of the data is improved.
In addition, the present invention also provides a computer-readable storage medium, where the computer-readable storage medium includes a data verification program, and when the data verification program is executed by a processor, the data verification program can implement the following operations:
an acquisition step: connecting the service systems corresponding to the data sources, and acquiring corresponding service data from each service system according to a preset time frequency;
a first configuration step: index configuration is carried out on each acquired service data to obtain indexes and index configuration information corresponding to each service data;
a second configuration step: carrying out rule configuration on indexes corresponding to the obtained business data to generate a configuration file, wherein the configuration file comprises index thresholds corresponding to the business data;
a calculation step: inputting the indexes and the index configuration information corresponding to the business data into a calculation engine for calculation to obtain the index values corresponding to the calculated business data; and
a checking step: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
The embodiment of the computer-readable storage medium of the present invention is substantially the same as the embodiments of the data verification method and the electronic device, and will not be described in detail herein.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (8)
1. A data verification method is applied to an electronic device, and is characterized by comprising the following steps:
an acquisition step: connecting the service systems corresponding to the data sources, and acquiring corresponding service data from each service system according to a preset time frequency;
a first configuration step: performing index configuration on each acquired service data to obtain indexes and index configuration information corresponding to each service data, where the index configuration information includes a data source associated with the indexes, an operation logic of the indexes, and configuration time, and the first configuration step specifically includes: performing index configuration on the service data according to the field attribute of the acquired service data to obtain indexes corresponding to each service data, associating the indexes corresponding to each service data, and matching the associated index configuration with a preset operation logic to obtain index configuration information;
a second configuration step: carrying out rule configuration on the obtained indexes corresponding to the service data to generate a configuration file, wherein the configuration file comprises index thresholds corresponding to the service data, and the index thresholds are generated according to the field attributes of the service data and the corresponding indexes during rule configuration;
a calculation step: inputting the indexes and the index configuration information corresponding to the business data into a calculation engine for calculation to obtain the index values corresponding to the calculated business data; and
a checking step: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
2. The data verification method of claim 1, wherein the verifying step comprises:
and when the index value corresponding to one piece of service data exceeds the corresponding index threshold value, judging that the service data is wrong, distributing a preset mark for the wrong service data and storing the mark to an early warning table.
3. The data verification method of claim 2, wherein after the verifying step further comprises:
early warning step: and scanning the early warning table at regular time, and triggering a preset early warning system to generate corresponding early warning information and sending the early warning information to a corresponding service system when the preset marked service data is scanned.
4. A data verification method as claimed in any one of claims 1 to 3, wherein said calculating step comprises:
finding out the operation logic of the index corresponding to each service data from the obtained index configuration information;
and carrying out SQL calculation on the indexes corresponding to the business data according to the found operation logic to obtain the index values corresponding to the business data.
5. An electronic device comprising a memory and a processor, the memory having stored therein a data verification program operable on the processor, the data verification program when executed by the processor implementing the steps of:
an acquisition step: connecting the service systems corresponding to the data sources, and acquiring corresponding service data from each service system according to a preset time frequency;
a first configuration step: performing index configuration on each acquired service data to obtain indexes and index configuration information corresponding to each service data, where the index configuration information includes a data source associated with the indexes, an operation logic of the indexes, and configuration time, and the first configuration step specifically includes: performing index configuration on the service data according to the field attribute of the acquired service data to obtain indexes corresponding to each service data, associating the indexes corresponding to each service data, and matching the associated index configuration with a preset operation logic to obtain index configuration information;
a second configuration step: carrying out rule configuration on the obtained indexes corresponding to the service data to generate a configuration file, wherein the configuration file comprises index thresholds corresponding to the service data, and the index thresholds are generated according to the field attributes of the service data and the corresponding indexes during rule configuration;
a calculation step: inputting the indexes and the index configuration information corresponding to the business data into a calculation engine for calculation to obtain the index values corresponding to the calculated business data; and
a checking step: and reading the configuration file by using a rule engine, analyzing the configuration file to obtain an index threshold value corresponding to each service data, and checking whether the index value corresponding to each service data exceeds the corresponding index threshold value.
6. The electronic device of claim 5, wherein the verifying step comprises:
and when the index value corresponding to one piece of service data exceeds the corresponding index threshold value, judging that the service data is wrong, distributing a preset mark for the wrong service data and storing the mark to an early warning table.
7. The electronic device of claim 6, wherein the verifying step further comprises:
early warning step: and scanning the early warning table at regular time, and triggering a preset early warning system to generate corresponding early warning information and sending the early warning information to a corresponding service system when the preset marked service data is scanned.
8. A computer-readable storage medium, comprising a data verification program, which when executed by a processor, implements the steps of the data verification method of any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910559406.6A CN110309125B (en) | 2019-06-24 | 2019-06-24 | Data verification method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910559406.6A CN110309125B (en) | 2019-06-24 | 2019-06-24 | Data verification method, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110309125A CN110309125A (en) | 2019-10-08 |
| CN110309125B true CN110309125B (en) | 2021-09-21 |
Family
ID=68076631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910559406.6A Active CN110309125B (en) | 2019-06-24 | 2019-06-24 | Data verification method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110309125B (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112733083B (en) * | 2019-10-28 | 2023-08-18 | 中国移动通信集团湖南有限公司 | Data verification method, system and device |
| CN111325466A (en) * | 2020-02-20 | 2020-06-23 | 深圳壹账通智能科技有限公司 | Intelligent early warning method and system |
| CN112540996B (en) * | 2020-11-12 | 2023-11-24 | Oppo（重庆）智能科技有限公司 | Service data verification method and device, electronic equipment and storage medium |
| CN112506901A (en) * | 2020-11-30 | 2021-03-16 | 深圳微众信用科技股份有限公司 | Data quality measuring method, device and medium |
| CN112445875B (en) * | 2020-12-14 | 2024-03-01 | 招商局金融科技有限公司 | Data association and verification method and device, electronic equipment and storage medium |
| CN112579376A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Double-run-stage data verification method, system, equipment and storage medium |
| CN112734358A (en) * | 2020-12-25 | 2021-04-30 | 山东浪潮通软信息科技有限公司 | Configuration method and device of business system |
| CN112767767B (en) * | 2021-01-29 | 2022-12-23 | 重庆子元科技有限公司 | Virtual training system |
| CN113128848A (en) * | 2021-04-01 | 2021-07-16 | 广州虎牙科技有限公司 | Data quality monitoring method for full-service index, electronic equipment and storage medium |
| CN113223200A (en) * | 2021-05-07 | 2021-08-06 | 广州天长信息技术有限公司 | Road stealing and escaping intelligent prevention and control method, storage medium and system based on index dimension |
| CN113553533A (en) * | 2021-06-10 | 2021-10-26 | 国网安徽省电力有限公司 | Index calculation method based on digital internal five-level market assessment system |
| CN113609283A (en) * | 2021-07-28 | 2021-11-05 | 浙江惠瀜网络科技有限公司 | Data acquisition method and system |
| CN113778535B (en) * | 2021-08-10 | 2024-02-23 | 济南浪潮数据技术有限公司 | Cloud host operation verification method, system, equipment and storage medium |
| CN113515923B (en) * | 2021-09-14 | 2022-01-04 | 浙江省金融综合服务平台管理有限公司 | Index processing implementation method applied to decision engine |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101135715A (en) * | 2007-05-25 | 2008-03-05 | 中兴通讯股份有限公司 | North direction interface data integrity verification method |
| CN105550511A (en) * | 2015-12-11 | 2016-05-04 | 北京锐软科技股份有限公司 | Data quality evaluation system and method based on data verification technique |
| CN106452933A (en) * | 2015-08-05 | 2017-02-22 | 腾讯科技（北京）有限公司 | Service data statistical method, device and system |
| CN107038200A (en) * | 2016-12-15 | 2017-08-11 | 平安科技（深圳）有限公司 | Business data processing method and system |
| CN109522318A (en) * | 2018-10-22 | 2019-03-26 | 中国银行股份有限公司 | A kind of data quality management method and system |
| CN109766370A (en) * | 2018-12-27 | 2019-05-17 | 口碑(上海)信息技术有限公司 | Data processing method, data service system and equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101719237A (en) * | 2009-12-09 | 2010-06-02 | 南京联创科技集团股份有限公司 | Data quality monitoring method based on full service indicator rule allocation |
-
2019
- 2019-06-24 CN CN201910559406.6A patent/CN110309125B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101135715A (en) * | 2007-05-25 | 2008-03-05 | 中兴通讯股份有限公司 | North direction interface data integrity verification method |
| CN106452933A (en) * | 2015-08-05 | 2017-02-22 | 腾讯科技（北京）有限公司 | Service data statistical method, device and system |
| CN105550511A (en) * | 2015-12-11 | 2016-05-04 | 北京锐软科技股份有限公司 | Data quality evaluation system and method based on data verification technique |
| CN107038200A (en) * | 2016-12-15 | 2017-08-11 | 平安科技（深圳）有限公司 | Business data processing method and system |
| CN109522318A (en) * | 2018-10-22 | 2019-03-26 | 中国银行股份有限公司 | A kind of data quality management method and system |
| CN109766370A (en) * | 2018-12-27 | 2019-05-17 | 口碑(上海)信息技术有限公司 | Data processing method, data service system and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110309125A (en) | 2019-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110309125B (en) | Data verification method, electronic device and storage medium | |
| CN108446407B (en) | Database auditing method and device based on block chain | |
| CN108427705B (en) | Electronic device, distributed system log query method and storage medium | |
| CN109474578B (en) | Message checking method, device, computer equipment and storage medium | |
| CN113242236A (en) | Method for constructing network entity threat map | |
| CN110599354A (en) | Online reconciliation method, system, computer device and computer-readable storage medium | |
| CN105260639A (en) | Face recognition system data update method and device | |
| CN112307747A (en) | Vehicle accessory retrieval method and device, computer equipment and readable storage medium | |
| CN109241163B (en) | Electronic certificate generation method and terminal equipment | |
| CN114328017A (en) | Database backup method, system, computer equipment and storage medium | |
| CN106095511A (en) | A kind of server updating method and apparatus | |
| CN112364022B (en) | Information deduction management method, device, computer equipment and readable storage medium | |
| CN110866007B (en) | Information management method, system and computer equipment for big data application and table | |
| CN110489416B (en) | Information storage method based on data processing and related equipment | |
| CN109271431B (en) | Data extraction method, device, computer equipment and storage medium | |
| CN109284833B (en) | Method, apparatus and storage medium for obtaining feature data for machine learning model | |
| CN116719799A (en) | Environment-friendly data management method, device, computer equipment and storage medium | |
| CN114168610B (en) | Distributed storage and query method and system based on line sequence division | |
| CN113259154B (en) | Method and device for informing middle station data verification, computer equipment and storage medium | |
| CN111708795B (en) | Object identification generation method, object identification updating device, computer equipment and medium | |
| CN112583761B (en) | Management method and device of security entity, computer equipment and storage medium | |
| CN114238767A (en) | Service recommendation method and device, computer equipment and storage medium | |
| CN110119337B (en) | Data analysis method and device and server | |
| CN112948370B (en) | Data classification method and device and computer equipment | |
| CN111131393A (en) | User activity data statistical method, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |